media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)
Subsystems (2)
v2 cover media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up system
Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware modules are:
- ACE (Algorithm Context Engine): This module is for algorithm computing
when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.
IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.
In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
Implementation:
There are two different drivers to handle ACE and CSI hardware modules inside IVSC.
- mei_csi: MEI client driver to send commands and receive notifications
from CSI module.
- mei_ace: MEI client driver to send commands and get status from ACE
module.
Interface is exposed via ivsc.h to acquire and release camera sensor and CSI-2 link.
Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
-----------------------------------------------------------------------------
| Host Processor |
| |
| ----------------- ----------------- --------------- |
| | | | | | | I2C |
| | IPU | | ISH | |camera driver|--| |
| | | | | | | | |
| ----------------- ----------------- --------------- | |
| | | | | |
| | | --------------- | |
| | | | | | |
| | | | IVSC driver | | |
| | | | | | |
| | | --------------- | |
| | | | | |
----------------|-----------------------|----------------------|---------|---
| CSI | I2C |SPI |
| | | |
----------------|-----------------------|---------------- | |
| IVSC | | | |
| | | | |
| ----------------- ----------------- | | |
| | | | | | | |
| | CSI | | ACE | |------| |
| | | | | | |
| ----------------- ----------------- | |
| | | I2C | |
----------------|-----------------------|---------------- |
| CSI | |
| | |
-------------------------------- |
| | I2C |
| camera sensor |-----------------------------|
| |
--------------------------------Wentong Wu (3):
media: pci: intel: ivsc: Add CSI submodule
media: pci: intel: ivsc: Add ACE submodule
media: pci: intel: ivsc: Add acquire/release API for ivsc
drivers/media/pci/Kconfig | 1 + drivers/media/pci/intel/Makefile | 2 + drivers/media/pci/intel/ivsc/Kconfig | 12 + drivers/media/pci/intel/ivsc/Makefile | 7 + drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++ drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++ drivers/media/pci/intel/ivsc/mei_ace.h | 36 ++ drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++ drivers/media/pci/intel/ivsc/mei_csi.h | 60 ++++ include/linux/ivsc.h | 74 ++++ 10 files changed, 1090 insertions(+) create mode 100644 drivers/media/pci/intel/ivsc/Kconfig create mode 100644 drivers/media/pci/intel/ivsc/Makefile create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h create mode 100644 include/linux/ivsc.h
Signature · 1 line
2.25.1
PATCH v2 1/3 media: pci: intel: ivsc: Add CSI submodule
CSI is a submodule of IVSC which can route camera sensor data to the outbound MIPI CSI-2 interface.
The interface communicating with firmware is via MEI. There is a separate MEI UUID, which this driver uses to enumerate.
To route camera sensor data to host, the caller specifies link frequency and number of data lanes. This information is sent to firmware by sending MEI command.
Two APIs are exported: csi_set_link_owner is used to switch ownership of CSI-2 link, and csi_set_link_cfg is to configure CSI-2 link when routing camera sensor data to host.
CSI also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both IVSC and host Image Processing Unit(IPU) can't get image data. And when this mode is turned on, host Image Processing Unit(IPU) driver is informed via the registered callback, so that user can be notified.
--- drivers/media/pci/Kconfig | 1 + drivers/media/pci/intel/Makefile | 2 + drivers/media/pci/intel/ivsc/Kconfig | 12 + drivers/media/pci/intel/ivsc/Makefile | 5 + drivers/media/pci/intel/ivsc/mei_csi.c | 342 +++++++++++++++++++++++++ drivers/media/pci/intel/ivsc/mei_csi.h | 60 +++++ include/linux/ivsc.h | 19 ++ 7 files changed, 441 insertions(+) create mode 100644 drivers/media/pci/intel/ivsc/Kconfig create mode 100644 drivers/media/pci/intel/ivsc/Makefile create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h create mode 100644 include/linux/ivsc.h
drivers/media/pci/Kconfig +1 -0
@@ -74,6 +74,7 @@ config VIDEO_PCI_SKELETON when developing new drivers. source "drivers/media/pci/intel/ipu3/Kconfig" +source "drivers/media/pci/intel/ivsc/Kconfig" endif #MEDIA_PCI_SUPPORT endif #PCI
drivers/media/pci/intel/Makefile +2 -0
@@ -4,3 +4,5 @@ # obj-y += ipu3/ + +obj-$(CONFIG_INTEL_VSC) += ivsc/
drivers/media/pci/intel/ivsc/Kconfig new file +12 -0
@@ -0,0 +1,12 @@ +# SPDX-License-Identifier: GPL-2.0-only +# Copyright (C) 2023, Intel Corporation. All rights reserved. + +config INTEL_VSC + tristate "Intel Visual Sensing Controller" + depends on INTEL_MEI + help + This adds support for Intel Visual Sensing Controller (IVSC). + + Enables the IVSC firmware services required for controlling + camera sensor ownership and CSI-2 link through Image Processing + Unit(IPU) driver of Intel.
drivers/media/pci/intel/ivsc/Makefile new file +5 -0
@@ -0,0 +1,5 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +# Copyright (C) 2023, Intel Corporation. All rights reserved. + +obj-$(CONFIG_INTEL_VSC) += mei_csi.o
drivers/media/pci/intel/ivsc/mei_csi.c new file +342 -0
342 lines not on this page · raw message
drivers/media/pci/intel/ivsc/mei_csi.h new file +60 -0
60 lines not on this page · raw message
include/linux/ivsc.h new file +19 -0
@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* +* Copyright (C) 2023 Intel Corporation. All rights reserved. +* Intel Visual Sensing Controller interface +*/ + +#ifndef _LINUX_IVSC_H_ +#define _LINUX_IVSC_H_ + +#include <linux/types.h> + +/* IVSC privacy status definition */ +enum ivsc_privacy_status { +IVSC_PRIVACY_OFF , +IVSC_PRIVACY_ON , +IVSC_PRIVACY_MAX , +}; + +#endif
Signature · 1 line
2.25.1
Sakari Ailus
maintainerRe: [PATCH v2 1/3] media: pci: intel: ivsc: Add CSI submodule
Hi Wentong,
Cc'ing Hans as well.
Quoted 22 lines · Wentong Wu
CSI is a submodule of IVSC which can route camera sensor data to the outbound MIPI CSI-2 interface.
The interface communicating with firmware is via MEI. There is a separate MEI UUID, which this driver uses to enumerate.
To route camera sensor data to host, the caller specifies link frequency and number of data lanes. This information is sent to firmware by sending MEI command.
Two APIs are exported:
csi_set_link_owneris used to switch ownership of CSI-2 link, andcsi_set_link_cfgis to configure CSI-2 link when routing camera sensor data to host.CSI also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both IVSC and host Image Processing Unit(IPU) can't get image data. And when this mode is turned on, host Image Processing Unit(IPU) driver is informed via the registered callback, so that user can be notified.
Signed-off-by Wentong Wu <[email protected]>
Quoted 8 lines
--- drivers/media/pci/Kconfig | 1 + drivers/media/pci/intel/Makefile | 2 + drivers/media/pci/intel/ivsc/Kconfig | 12 + drivers/media/pci/intel/ivsc/Makefile | 5 + drivers/media/pci/intel/ivsc/mei_csi.c | 342 +++++++++++++++++++++++++ drivers/media/pci/intel/ivsc/mei_csi.h | 60 +++++ include/linux/ivsc.h | 19 ++
Please add a MAINTAINERS entry for this (ivsc dir).
Quoted 64 lines
7 files changed, 441 insertions(+) create mode 100644 drivers/media/pci/intel/ivsc/Kconfig create mode 100644 drivers/media/pci/intel/ivsc/Makefile create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h create mode 100644 include/linux/ivsc.hdrivers/media/pci/Kconfig +1
-0@@ -74,6 +74,7 @@ config VIDEO_PCI_SKELETON when developing new drivers. source "drivers/media/pci/intel/ipu3/Kconfig" +source "drivers/media/pci/intel/ivsc/Kconfig" endif #MEDIA_PCI_SUPPORT endif #PCIdrivers/media/pci/intel/Makefile +2
-0@@ -4,3 +4,5 @@ # obj-y += ipu3/ + +obj-$(CONFIG_INTEL_VSC) += ivsc/drivers/media/pci/intel/ivsc/Kconfig new file +12
-0@@ -0,0 +1,12 @@ +# SPDX-License-Identifier: GPL-2.0-only +# Copyright (C) 2023, Intel Corporation. All rights reserved. + +config INTEL_VSC + tristate "Intel Visual Sensing Controller" + depends on INTEL_MEI + help + This adds support for Intel Visual Sensing Controller (IVSC). + + Enables the IVSC firmware services required for controlling + camera sensor ownership and CSI-2 link through Image Processing + Unit(IPU) driver of Intel.drivers/media/pci/intel/ivsc/Makefile new file +5
-0@@ -0,0 +1,5 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +# Copyright (C) 2023, Intel Corporation. All rights reserved. + +obj-$(CONFIG_INTEL_VSC) += mei_csi.odrivers/media/pci/intel/ivsc/mei_csi.c new file +0
-0
drivers/media/pci/intel/ivsc/mei_csi.c line 3
Quoted 3 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +3
-0@@ -0,0 +1,342 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright (C) 2023 Intel Corporation. All rights reserved.
Do you still need the "all rights reserved" part?
drivers/media/pci/intel/ivsc/mei_csi.c line 7
Quoted 4 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +4
-0+* Intel Visual Sensing Controller CSI Linux driver +*/ + +/*
You can merge the two comments.
Quoted 137 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +137
-0+* To set ownership of CSI-2 link and to configure CSI-2 link, there +* are specific commands, which are sent via MEI protocol. The send +* command function uses "completion" as a synchronization mechanism. +* The response for command is received via a mei callback which wakes +* up the caller. There can be only one outstanding command at a time. +*/ + +#include <linux/completion.h> +#include <linux/delay.h> +#include <linux/kernel.h> +#include <linux/math64.h> +#include <linux/mei_cl_bus.h> +#include <linux/module.h> +#include <linux/mutex.h> +#include <linux/slab.h> +#include <linux/units.h> +#include <linux/uuid.h> +#include <linux/workqueue.h> + +#include "mei_csi.h" + +#define MEI_CSI_DRIVER_NAME "ivsc_csi" + +/* the 5s used here is based on experiment */ +#define CSI_CMD_TIMEOUT (5 * HZ) +/* to setup CSI-2 link an extra delay needed and determined experimentally */ +#define CSI_FW_READY_DELAY_MS 100 +/* link frequency unit is 100kHz */ +#define CSI_LINK_FREQ (x) ((u32)(div_u64(x, 100 * HZ_PER_KHZ))) + +/* +* identify the command id supported by firmware +* IPC, as well as the privacy notification id +* used when processing privacy event. +*/ +enum csi_cmd_id { +/* used to set csi ownership */ +CSI_SET_OWNER =0 , + +/* used to configure CSI-2 link */ +CSI_SET_CONF =2 , + +/* privacy notification id used when privacy state changes */ +CSI_PRIVACY_NOTIF =6 , +}; + +/* configuration of the CSI-2 link between host and IVSC */ +struct csi_link_cfg { +/* number of data lanes used on the CSI-2 link */ +u32 nr_of_lanes; + +/* frequency of the CSI-2 link */ +u32 link_freq; + +/* for future use */ +u32 rsvd[2 ]; +} __packed; + +/* CSI command structure */ +struct csi_cmd { +u32 cmd_id; +union _cmd_param { +u32 param; +struct csi_link_cfg conf; + } param; +} __packed; + +/* CSI notification structure */ +struct csi_notif { +u32 cmd_id; +int status; +union _resp_cont { +u32 cont; +struct csi_link_cfg conf; + } cont; +} __packed; + +struct mei_csi { +struct mei_cl_device *cldev; + +/* command response */ +struct csi_notif cmd_response; +/* used to wait for command response from firmware */ +struct completion cmd_completion; + +/* work element used to handle firmware event */ +struct work_struct event_work; + +/* privacy status */ +enum ivsc_privacy_status status; +/* privacy callback */ +void (*callback)(void *,enum ivsc_privacy_status ); +/* privacy callback runtime context */ +void *context; +}; + +/* only one for now */ +static struct mei_csi *csi; +/* lock used to prevent multiple call to csi */ +static DEFINE_MUTEX (csi_mutex); + +/* send a command to firmware and mutex must be held by caller */ +static int mei_csi_send (u8 *buf,size_t len) +{ +struct csi_cmd *cmd = (struct csi_cmd *)buf; +int ret; + +reinit_completion (&csi->cmd_completion); + + ret =mei_cldev_send (csi->cldev, buf, len); +if (ret <0 ) + goto out; + + ret =wait_for_completion_killable_timeout (&csi->cmd_completion, +CSI_CMD_TIMEOUT ); +if (ret <0 ) { + goto out; + }else if (!ret) { + ret = -ETIMEDOUT ; + goto out; + } + +/* command response status */ + ret = csi->cmd_response.status; +if (ret) { + ret = -EINVAL ; + goto out; + } + +if (csi->cmd_response.cmd_id != cmd->cmd_id) + ret = -EINVAL ; + +out: +return ret; +} + +int csi_set_link_owner(enum csi_link_owner owner,
drivers/media/pci/intel/ivsc/mei_csi.c line 154
Quoted 10 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +10
-0+void (*callback)(void *,enum ivsc_privacy_status ), + void *context) +{ + struct csi_cmd cmd = { 0 }; +size_t cmd_size; +int ret; + + cmd.cmd_id =CSI_SET_OWNER ; + cmd.param.param = owner; + cmd_size =sizeof (cmd.cmd_id) +sizeof (cmd.param.param);
You could initialise these in variable declaration.
Quoted 98 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +98
-0+ +mutex_lock (&csi_mutex); +if (unlikely (!csi)) { +mutex_unlock (&csi_mutex); +return -EAGAIN ; + } + + ret =mei_csi_send ((u8 *)&cmd, cmd_size); + +/* +* cancel possible event work and wait for it to finish +* to make sure no work running for the ongoing changes +* to callback and context. +*/ +cancel_work_sync (&csi->event_work); + csi->callback = callback; + csi->context = context; +mutex_unlock (&csi_mutex); + +return ret; +} +EXPORT_SYMBOL_NS_GPL (csi_set_link_owner,IVSC ); + +int csi_set_link_cfg (u32 nr_of_lanes,u64 link_freq) +{ +struct csi_cmd cmd = {0 }; +size_t cmd_size; +int ret; + + cmd.cmd_id =CSI_SET_CONF ; + cmd.param.conf.nr_of_lanes = nr_of_lanes; + cmd.param.conf.link_freq =CSI_LINK_FREQ (link_freq); + cmd_size =sizeof (cmd.cmd_id) +sizeof (cmd.param.conf); + +mutex_lock (&csi_mutex); +if (unlikely (!csi)) { +mutex_unlock (&csi_mutex); +return -EAGAIN ; + } + + ret =mei_csi_send ((u8 *)&cmd, cmd_size); +/* +* wait configuration ready if download success. placing +* delay under mutex is to make sure current command flow +* completed before starting a possible new one. +*/ +if (!ret) +msleep (CSI_FW_READY_DELAY_MS ); +mutex_unlock (&csi_mutex); + +return ret; +} +EXPORT_SYMBOL_NS_GPL (csi_set_link_cfg,IVSC ); + +/* event handling routine */ +static void mei_csi_event_work (struct work_struct *work) +{ +if (csi->callback) + csi->callback (csi->context,READ_ONCE (csi->status)); +} + +/* callback for receive */ +static void mei_csi_rx (struct mei_cl_device *cldev) +{ +struct csi_notif notif = {0 }; +int ret; + + ret =mei_cldev_recv (cldev, (u8 *)¬if,sizeof (notif)); +if (ret <0 ) { +dev_err (&cldev->dev,"recv error: %d\n" , ret); +return ; + } + +switch (notif.cmd_id) { +case CSI_PRIVACY_NOTIF : +if (notif.cont.cont <IVSC_PRIVACY_MAX ) { +WRITE_ONCE (csi->status, notif.cont.cont); + +schedule_work (&csi->event_work); + } +break ; +case CSI_SET_OWNER : +case CSI_SET_CONF : +memcpy (&csi->cmd_response, ¬if, ret); + +complete (&csi->cmd_completion); +break ; +default : +break ; + } +} + +static int mei_csi_probe (struct mei_cl_device *cldev, +const struct mei_cl_device_id *id) +{ +int ret; + +mutex_lock (&csi_mutex);
drivers/media/pci/intel/ivsc/mei_csi.c line 262
Quoted 10 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +10
-0+/* +* only instance is possible in the current hardware, +* but adding protection for future hardware. +*/ +if (csi) { + ret = -EBUSY ; + goto err_unlock; + } + + csi =kzalloc (sizeof (*csi),GFP_KERNEL );
How do you know the MEI CSI and the MEI ACE devices are related to a given camera sensor? I'd expect this information to come from system firmware but I don't see this being done in this patchset.
Ideally there should be also a way to obtain the appropriate MEI CSI and MEI ACE devices without assuming there's a single one in the system.
Quoted 171 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +80
-0+if (!csi) { + ret = -ENOMEM ; + goto err_unlock; + } + + csi->cldev = cldev; +init_completion (&csi->cmd_completion); +INIT_WORK (&csi->event_work, mei_csi_event_work); + +mei_cldev_set_drvdata (cldev, csi); + + ret =mei_cldev_enable (cldev); +if (ret <0 ) { +dev_err (&cldev->dev,"mei_cldev_enable failed: %d\n" , ret); + goto err_free; + } + + ret =mei_cldev_register_rx_cb (cldev, mei_csi_rx); +if (ret) { +dev_err (&cldev->dev,"event cb registration failed: %d\n" , ret); + goto err_disable; + } +mutex_unlock (&csi_mutex); + +return 0 ; + +err_disable: +mei_cldev_disable (cldev); + +err_free: +kfree (csi); +/* disable csi */ + csi =NULL ; + +err_unlock: +mutex_unlock (&csi_mutex); + +return ret; +} + +static void mei_csi_remove (struct mei_cl_device *cldev) +{ +mutex_lock (&csi_mutex); +/* disable mei csi client device */ +mei_cldev_disable (cldev); + +/* cancel event work and wait for it to finish */ +cancel_work_sync (&csi->event_work); + +kfree (csi); +/* disable csi */ + csi =NULL ; +mutex_unlock (&csi_mutex); +} + +#define MEI_CSI_UUID UUID_LE(0x92335FCF, 0x3203, 0x4472, \ + 0xAF, 0x93, 0x7b, 0x44, 0x53, 0xAC, 0x29, 0xDA) + +static const struct mei_cl_device_id mei_csi_tbl[] = { + {MEI_CSI_DRIVER_NAME ,MEI_CSI_UUID ,MEI_CL_VERSION_ANY }, + +/* required last entry */ + { } +}; +MODULE_DEVICE_TABLE (mei, mei_csi_tbl); + +static struct mei_cl_driver mei_csi_driver = { + .id_table = mei_csi_tbl, + .name =MEI_CSI_DRIVER_NAME , + + .probe = mei_csi_probe, + .remove = mei_csi_remove, +}; + +module_mei_cl_driver (mei_csi_driver); + +MODULE_AUTHOR ("Wentong Wu <[email protected]>" ); +MODULE_AUTHOR ("Zhifeng Wang <[email protected]>" ); +MODULE_DESCRIPTION ("Device driver for IVSC CSI" ); +MODULE_LICENSE ("GPL" );drivers/media/pci/intel/ivsc/mei_csi.h new file +60
-0@@ -0,0 +1,60 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* +* Copyright (C) 2023 Intel Corporation. All rights reserved. +* Intel Visual Sensing Controller CSI external interface +*/ + +#ifndef _MEI_CSI_H_ +#define _MEI_CSI_H_ + +#include <linux/ivsc.h> + +/* CSI-2 link ownership definition */ +enum csi_link_owner { +CSI_LINK_IVSC , +CSI_LINK_HOST , +}; + +#if IS_ENABLED (CONFIG_INTEL_VSC ) +/* +* @brief set CSI-2 link ownership +* +* @param owner The csi ownership being set +* @param callback The pointer of privacy callback being set +* @param context Privacy callback runtime context +* +* @return 0 on success, negative on failure +*/ +int csi_set_link_owner (enum csi_link_owner owner, +void (*callback)(void *,enum ivsc_privacy_status ), +void *context); + +/* +* @brief configure CSI-2 link between host and IVSC +* with provided parameters +* +* @param nr_of_lanes The number of data lanes being used +* on the CSI-2 link +* @param link_freq The frequency being set on the CSI-2 link +* +* @return 0 on success, negative on failure +*/ +int csi_set_link_cfg (u32 nr_of_lanes,u64 link_freq); + +#else +static inline +int csi_set_link_owner (enum csi_link_owner owner, +void (*callback)(void *,enum ivsc_privacy_status ), +void *context) +{ +return 0 ; +} + +static inline int csi_set_link_cfg (u32 nr_of_lanes,u64 link_freq) +{ +return 0 ; +} + +#endif + +#endif include/linux/ivsc.h new file +19
-0@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* +* Copyright (C) 2023 Intel Corporation. All rights reserved. +* Intel Visual Sensing Controller interface +*/ + +#ifndef _LINUX_IVSC_H_ +#define _LINUX_IVSC_H_ + +#include <linux/types.h> + +/* IVSC privacy status definition */ +enum ivsc_privacy_status { +IVSC_PRIVACY_OFF , +IVSC_PRIVACY_ON , +IVSC_PRIVACY_MAX , +}; + +#endif
Signature · 3 lines
Kind regards, Sakari Ailus
PATCH v2 2/3 media: pci: intel: ivsc: Add ACE submodule
In reply to Wentong WuACE is a submodule of IVSC which controls camera sensor's ownership, belonging to host or IVSC. When IVSC owns camera sensor, it is for algorithm computing. When host wants to control camera sensor, ACE module needs to be informed of ownership with defined interface.
The interface is via MEI. There is a separate MEI UUID, which this driver uses to enumerate.
To switch ownership of camera sensor between IVSC and host, the caller specifies the defined ownership information which will be sent to firmware by sending MEI command.
The only exported API ace_set_camera_owner is to switch ownership of camera sensor.
--- drivers/media/pci/intel/ivsc/Makefile | 1 + drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++ drivers/media/pci/intel/ivsc/mei_ace.h | 36 ++ 3 files changed, 509 insertions(+) create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
drivers/media/pci/intel/ivsc/Makefile +1 -0
@@ -3,3 +3,4 @@ # Copyright (C) 2023, Intel Corporation. All rights reserved. obj-$(CONFIG_INTEL_VSC) += mei_csi.o +obj-$(CONFIG_INTEL_VSC) += mei_ace.o
drivers/media/pci/intel/ivsc/mei_ace.c new file +472 -0
472 lines not on this page · raw message
drivers/media/pci/intel/ivsc/mei_ace.h new file +36 -0
@@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* +* Copyright (C) 2023 Intel Corporation. All rights reserved. +* Intel Visual Sensing Controller ACE external interface +*/ + +#ifndef _MEI_ACE_H_ +#define _MEI_ACE_H_ + +#include <linux/types.h> + +/* identify camera sensor ownership */ +enum ace_camera_owner { +ACE_CAMERA_IVSC , +ACE_CAMERA_HOST , +}; + +#if IS_ENABLED (CONFIG_INTEL_VSC ) +/* +* @brief set camera sensor ownership +* +* @param owner Camera sensor ownership being set +* +* @return 0 on success, negative on failure +*/ +int ace_set_camera_owner (enum ace_camera_owner owner); + +#else +static inline int ace_set_camera_owner (enum ace_camera_owner owner) +{ +return 0 ; +} + +#endif + +#endif
Signature · 1 line
2.25.1
PATCH v2 3/3 media: pci: intel: ivsc: Add acquire/release API for ivsc
In reply to Wentong WuIVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.
This commit defines the interfaces between IVSC and camera sensor driver in include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.
--- drivers/media/pci/intel/ivsc/Makefile | 1 + drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++++++++++++++++++++++++ include/linux/ivsc.h | 55 ++++++++++++++++++ 3 files changed, 140 insertions(+) create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
drivers/media/pci/intel/ivsc/Makefile +1 -0
@@ -4,3 +4,4 @@ obj-$(CONFIG_INTEL_VSC) += mei_csi.o obj-$(CONFIG_INTEL_VSC) += mei_ace.o +obj-$(CONFIG_INTEL_VSC) += ivsc.o
drivers/media/pci/intel/ivsc/ivsc.c new file +84 -0
84 lines not on this page · raw message
include/linux/ivsc.h +55 -0
59 lines not on this page · raw message
Signature · 1 line
2.25.1
Sakari Ailus
maintainerRe: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc
Hi Wentong,
Thanks for the patchset.
Quoted 9 lines · Wentong Wu
IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.
Quoted 5 lines
This commit defines the interfaces between IVSC and camera sensor driver in
include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.
The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image pipeline.
I suppose the intended use cases assume a single instance of IVSC (as well as MEI) but there can, and often are, be multiple camera sensors in the system. The decision whether to request pass-through from IVCS can't be done in the camera sensor driver, and should not be visible to the camera sensor driver. Exposing IVSC as a V4L2 sub-device makes this trivial to address, as the IVSC driver's V4L2 sub-device video s_stream() operation gets called before streaming is started.
The information whether IVSC is found between the camera sensor and the host's CSI-2 receiver (IPU in this case) should come from system firmware and accessed most probably by what is called cio2-bridge at the moment.
The privacy status can be a V4L2 control.
Also cc Hans.
Quoted 172 lines
Signed-off-by Wentong Wu <[email protected]>--- drivers/media/pci/intel/ivsc/Makefile | 1 + drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++++++++++++++++++++++++ include/linux/ivsc.h | 55 ++++++++++++++++++ 3 files changed, 140 insertions(+) create mode 100644 drivers/media/pci/intel/ivsc/ivsc.cdrivers/media/pci/intel/ivsc/Makefile +1
-0@@ -4,3 +4,4 @@ obj-$(CONFIG_INTEL_VSC) += mei_csi.o obj-$(CONFIG_INTEL_VSC) += mei_ace.o +obj-$(CONFIG_INTEL_VSC) += ivsc.odrivers/media/pci/intel/ivsc/ivsc.c new file +84
-0@@ -0,0 +1,84 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright (C) 2023 Intel Corporation. All rights reserved. +* Intel Visual Sensing Controller interface +*/ + +#include <linux/delay.h> +#include <linux/ivsc.h> +#include <linux/module.h> +#include <linux/mutex.h> + +#include "mei_ace.h" +#include "mei_csi.h" + +/* lock for ivsc APIs */ +static DEFINE_MUTEX (ivsc_mutex); + +int ivsc_acquire_camera (u32 nr_of_lanes,u64 link_freq, +void (*callback)(void *,enum ivsc_privacy_status ), +void *context) +{ +int ret; + +mutex_lock (&ivsc_mutex); + +/* switch camera sensor ownership to host */ + ret =ace_set_camera_owner (ACE_CAMERA_HOST ); +if (ret) + goto error; + +/* switch CSI-2 link to host */ + ret =csi_set_link_owner (CSI_LINK_HOST , callback, context); +if (ret) + goto release_camera; + +/* configure CSI-2 link */ + ret =csi_set_link_cfg (nr_of_lanes, link_freq); +if (ret) + goto release_csi; + +mutex_unlock (&ivsc_mutex); + +return 0 ; + +release_csi: +csi_set_link_owner (CSI_LINK_IVSC ,NULL ,NULL ); + +release_camera: +ace_set_camera_owner (ACE_CAMERA_IVSC ); + +error: +mutex_unlock (&ivsc_mutex); + +return ret; +} +EXPORT_SYMBOL_GPL (ivsc_acquire_camera); + +int ivsc_release_camera (void ) +{ +int ret; + +mutex_lock (&ivsc_mutex); + +/* switch CSI-2 link to IVSC */ + ret =csi_set_link_owner (CSI_LINK_IVSC ,NULL ,NULL ); +if (ret) + goto error; + +/* switch camera sensor ownership to IVSC */ + ret =ace_set_camera_owner (ACE_CAMERA_IVSC ); + +error: +mutex_unlock (&ivsc_mutex); + +return ret; +} +EXPORT_SYMBOL_GPL (ivsc_release_camera); + +MODULE_AUTHOR ("Wentong Wu <[email protected]>" ); +MODULE_AUTHOR ("Zhifeng Wang <[email protected]>" ); +MODULE_SOFTDEP ("pre: mei_csi mei_ace" ); +MODULE_DESCRIPTION ("IVSC interface" ); +MODULE_LICENSE ("GPL" ); +MODULE_IMPORT_NS (IVSC );include/linux/ivsc.h +55
-0@@ -16,4 +16,59 @@ enum ivsc_privacy_status { IVSC_PRIVACY_MAX, }; +#if IS_ENABLED (CONFIG_INTEL_VSC ) +/* +* @brief Acquire camera sensor ownership to host and setup +* the CSI-2 link between host and IVSC +* +* IVSC provides a privacy mode. When the privacy mode is turned +* on, camera sensor can't be used. This means that both IVSC and +* host Image Processing Unit(IPU) can't get image data. And when +* this mode is turned on, host Image Processing Unit(IPU) driver +* is informed via the registered callback, so that user can be +* notified. +* +* @param nr_of_lanes Number of data lanes used on the CSI-2 link +* @param link_freq Frequency of the CSI-2 link +* @param callback The pointer of privacy callback function +* @param context Privacy callback function runtime context +* +* @retval 0 If success +* @retval -EIO IO error +* @retval -EINVAL Invalid argument +* @retval -EAGAIN IVSC device not ready +* @retval negative values for other errors +*/ +int ivsc_acquire_camera (u32 nr_of_lanes,u64 link_freq, +void (*callback)(void *,enum ivsc_privacy_status ), +void *context); + +/* +* @brief Release camera sensor ownership and stop the CSI-2 +* link between host and IVSC +* +* @retval 0 If success +* @retval -EIO IO error +* @retval -EINVAL Invalid argument +* @retval -EAGAIN IVSC device not ready +* @retval negative values for other errors +*/ +int ivsc_release_camera (void ); + +#else +static inline +int ivsc_acquire_camera (u32 nr_of_lanes,u64 link_freq, +void (*callback)(void *,enum ivsc_privacy_status ), +void *context) +{ +return 0 ; +} + +static inline int ivsc_release_camera (void ) +{ +return 0 ; +} + +#endif +#endif
Signature · 3 lines
Kind regards, Sakari Ailus
Re: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc
Hi Sakari,
Quoted 20 lines · Sakari Ailus
Hi Wentong,
Thanks for the patchset.
Quoted 9 lines · Wentong Wu
IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.
Quoted 5 lines
This commit defines the interfaces between IVSC and camera sensor driver in
include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.
Quoted 4 lines
The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image pipeline.
Yes I happened to discuss this exact same thing with Laurent at FOSDEM and we also came to the conclusion that the IVSC chip should be modeled as a V4L2 sub-device.
Regards,
Hans
Quoted 190 lines
I suppose the intended use cases assume a single instance of IVSC (as well as MEI) but there can, and often are, be multiple camera sensors in the system. The decision whether to request pass-through from IVCS can't be done in the camera sensor driver, and should not be visible to the camera sensor driver. Exposing IVSC as a V4L2 sub-device makes this trivial to address, as the IVSC driver's V4L2 sub-device video
s_stream()operation gets called before streaming is started.The information whether IVSC is found between the camera sensor and the host's CSI-2 receiver (IPU in this case) should come from system firmware and accessed most probably by what is called cio2-bridge at the moment.
The privacy status can be a V4L2 control.
Also cc Hans.
Quoted 172 lines
Signed-off-by Wentong Wu <[email protected]>--- drivers/media/pci/intel/ivsc/Makefile | 1 + drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++++++++++++++++++++++++ include/linux/ivsc.h | 55 ++++++++++++++++++ 3 files changed, 140 insertions(+) create mode 100644 drivers/media/pci/intel/ivsc/ivsc.cdrivers/media/pci/intel/ivsc/Makefile +1
-0@@ -4,3 +4,4 @@ obj-$(CONFIG_INTEL_VSC) += mei_csi.o obj-$(CONFIG_INTEL_VSC) += mei_ace.o +obj-$(CONFIG_INTEL_VSC) += ivsc.odrivers/media/pci/intel/ivsc/ivsc.c new file +84
-0@@ -0,0 +1,84 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright (C) 2023 Intel Corporation. All rights reserved. +* Intel Visual Sensing Controller interface +*/ + +#include <linux/delay.h> +#include <linux/ivsc.h> +#include <linux/module.h> +#include <linux/mutex.h> + +#include "mei_ace.h" +#include "mei_csi.h" + +/* lock for ivsc APIs */ +static DEFINE_MUTEX (ivsc_mutex); + +int ivsc_acquire_camera (u32 nr_of_lanes,u64 link_freq, +void (*callback)(void *,enum ivsc_privacy_status ), +void *context) +{ +int ret; + +mutex_lock (&ivsc_mutex); + +/* switch camera sensor ownership to host */ + ret =ace_set_camera_owner (ACE_CAMERA_HOST ); +if (ret) + goto error; + +/* switch CSI-2 link to host */ + ret =csi_set_link_owner (CSI_LINK_HOST , callback, context); +if (ret) + goto release_camera; + +/* configure CSI-2 link */ + ret =csi_set_link_cfg (nr_of_lanes, link_freq); +if (ret) + goto release_csi; + +mutex_unlock (&ivsc_mutex); + +return 0 ; + +release_csi: +csi_set_link_owner (CSI_LINK_IVSC ,NULL ,NULL ); + +release_camera: +ace_set_camera_owner (ACE_CAMERA_IVSC ); + +error: +mutex_unlock (&ivsc_mutex); + +return ret; +} +EXPORT_SYMBOL_GPL (ivsc_acquire_camera); + +int ivsc_release_camera (void ) +{ +int ret; + +mutex_lock (&ivsc_mutex); + +/* switch CSI-2 link to IVSC */ + ret =csi_set_link_owner (CSI_LINK_IVSC ,NULL ,NULL ); +if (ret) + goto error; + +/* switch camera sensor ownership to IVSC */ + ret =ace_set_camera_owner (ACE_CAMERA_IVSC ); + +error: +mutex_unlock (&ivsc_mutex); + +return ret; +} +EXPORT_SYMBOL_GPL (ivsc_release_camera); + +MODULE_AUTHOR ("Wentong Wu <[email protected]>" ); +MODULE_AUTHOR ("Zhifeng Wang <[email protected]>" ); +MODULE_SOFTDEP ("pre: mei_csi mei_ace" ); +MODULE_DESCRIPTION ("IVSC interface" ); +MODULE_LICENSE ("GPL" ); +MODULE_IMPORT_NS (IVSC );include/linux/ivsc.h +55
-0@@ -16,4 +16,59 @@ enum ivsc_privacy_status { IVSC_PRIVACY_MAX, }; +#if IS_ENABLED (CONFIG_INTEL_VSC ) +/* +* @brief Acquire camera sensor ownership to host and setup +* the CSI-2 link between host and IVSC +* +* IVSC provides a privacy mode. When the privacy mode is turned +* on, camera sensor can't be used. This means that both IVSC and +* host Image Processing Unit(IPU) can't get image data. And when +* this mode is turned on, host Image Processing Unit(IPU) driver +* is informed via the registered callback, so that user can be +* notified. +* +* @param nr_of_lanes Number of data lanes used on the CSI-2 link +* @param link_freq Frequency of the CSI-2 link +* @param callback The pointer of privacy callback function +* @param context Privacy callback function runtime context +* +* @retval 0 If success +* @retval -EIO IO error +* @retval -EINVAL Invalid argument +* @retval -EAGAIN IVSC device not ready +* @retval negative values for other errors +*/ +int ivsc_acquire_camera (u32 nr_of_lanes,u64 link_freq, +void (*callback)(void *,enum ivsc_privacy_status ), +void *context); + +/* +* @brief Release camera sensor ownership and stop the CSI-2 +* link between host and IVSC +* +* @retval 0 If success +* @retval -EIO IO error +* @retval -EINVAL Invalid argument +* @retval -EAGAIN IVSC device not ready +* @retval negative values for other errors +*/ +int ivsc_release_camera (void ); + +#else +static inline +int ivsc_acquire_camera (u32 nr_of_lanes,u64 link_freq, +void (*callback)(void *,enum ivsc_privacy_status ), +void *context) +{ +return 0 ; +} + +static inline int ivsc_release_camera (void ) +{ +return 0 ; +} + +#endif +#endif
Re: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc
Quoted 23 lines · Hans de Goede
Hi Sakari,
Quoted 20 lines · Sakari Ailus
Hi Wentong,
Thanks for the patchset.
Quoted 14 lines · Wentong Wu
IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.
This commit defines the interfaces between IVSC and camera sensor driver in
include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.
Quoted 8 lines
Quoted 4 lines
The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image pipeline.
Yes I happened to discuss this exact same thing with Laurent at FOSDEM and we also came to the conclusion that the IVSC chip should be modeled as a V4L2 sub-device.
Agreed.
Quoted 188 lines
Quoted 188 lines
I suppose the intended use cases assume a single instance of IVSC (as well as MEI) but there can, and often are, be multiple camera sensors in the system. The decision whether to request pass-through from IVCS can't be done in the camera sensor driver, and should not be visible to the camera sensor driver. Exposing IVSC as a V4L2 sub-device makes this trivial to address, as the IVSC driver's V4L2 sub-device video
s_stream()operation gets called before streaming is started.The information whether IVSC is found between the camera sensor and the host's CSI-2 receiver (IPU in this case) should come from system firmware and accessed most probably by what is called cio2-bridge at the moment.
The privacy status can be a V4L2 control.
Also cc Hans.
Quoted 172 lines
Signed-off-by Wentong Wu <[email protected]>--- drivers/media/pci/intel/ivsc/Makefile | 1 + drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++++++++++++++++++++++++ include/linux/ivsc.h | 55 ++++++++++++++++++ 3 files changed, 140 insertions(+) create mode 100644 drivers/media/pci/intel/ivsc/ivsc.cdrivers/media/pci/intel/ivsc/Makefile +1
-0@@ -4,3 +4,4 @@ obj-$(CONFIG_INTEL_VSC) += mei_csi.o obj-$(CONFIG_INTEL_VSC) += mei_ace.o +obj-$(CONFIG_INTEL_VSC) += ivsc.odrivers/media/pci/intel/ivsc/ivsc.c new file +84
-0@@ -0,0 +1,84 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright (C) 2023 Intel Corporation. All rights reserved. +* Intel Visual Sensing Controller interface +*/ + +#include <linux/delay.h> +#include <linux/ivsc.h> +#include <linux/module.h> +#include <linux/mutex.h> + +#include "mei_ace.h" +#include "mei_csi.h" + +/* lock for ivsc APIs */ +static DEFINE_MUTEX (ivsc_mutex); + +int ivsc_acquire_camera (u32 nr_of_lanes,u64 link_freq, +void (*callback)(void *,enum ivsc_privacy_status ), +void *context) +{ +int ret; + +mutex_lock (&ivsc_mutex); + +/* switch camera sensor ownership to host */ + ret =ace_set_camera_owner (ACE_CAMERA_HOST ); +if (ret) + goto error; + +/* switch CSI-2 link to host */ + ret =csi_set_link_owner (CSI_LINK_HOST , callback, context); +if (ret) + goto release_camera; + +/* configure CSI-2 link */ + ret =csi_set_link_cfg (nr_of_lanes, link_freq); +if (ret) + goto release_csi; + +mutex_unlock (&ivsc_mutex); + +return 0 ; + +release_csi: +csi_set_link_owner (CSI_LINK_IVSC ,NULL ,NULL ); + +release_camera: +ace_set_camera_owner (ACE_CAMERA_IVSC ); + +error: +mutex_unlock (&ivsc_mutex); + +return ret; +} +EXPORT_SYMBOL_GPL (ivsc_acquire_camera); + +int ivsc_release_camera (void ) +{ +int ret; + +mutex_lock (&ivsc_mutex); + +/* switch CSI-2 link to IVSC */ + ret =csi_set_link_owner (CSI_LINK_IVSC ,NULL ,NULL ); +if (ret) + goto error; + +/* switch camera sensor ownership to IVSC */ + ret =ace_set_camera_owner (ACE_CAMERA_IVSC ); + +error: +mutex_unlock (&ivsc_mutex); + +return ret; +} +EXPORT_SYMBOL_GPL (ivsc_release_camera); + +MODULE_AUTHOR ("Wentong Wu <[email protected]>" ); +MODULE_AUTHOR ("Zhifeng Wang <[email protected]>" ); +MODULE_SOFTDEP ("pre: mei_csi mei_ace" ); +MODULE_DESCRIPTION ("IVSC interface" ); +MODULE_LICENSE ("GPL" ); +MODULE_IMPORT_NS (IVSC );include/linux/ivsc.h +55
-0@@ -16,4 +16,59 @@ enum ivsc_privacy_status { IVSC_PRIVACY_MAX, }; +#if IS_ENABLED (CONFIG_INTEL_VSC ) +/* +* @brief Acquire camera sensor ownership to host and setup +* the CSI-2 link between host and IVSC +* +* IVSC provides a privacy mode. When the privacy mode is turned +* on, camera sensor can't be used. This means that both IVSC and +* host Image Processing Unit(IPU) can't get image data. And when +* this mode is turned on, host Image Processing Unit(IPU) driver +* is informed via the registered callback, so that user can be +* notified. +* +* @param nr_of_lanes Number of data lanes used on the CSI-2 link +* @param link_freq Frequency of the CSI-2 link +* @param callback The pointer of privacy callback function +* @param context Privacy callback function runtime context +* +* @retval 0 If success +* @retval -EIO IO error +* @retval -EINVAL Invalid argument +* @retval -EAGAIN IVSC device not ready +* @retval negative values for other errors +*/ +int ivsc_acquire_camera (u32 nr_of_lanes,u64 link_freq, +void (*callback)(void *,enum ivsc_privacy_status ), +void *context); + +/* +* @brief Release camera sensor ownership and stop the CSI-2 +* link between host and IVSC +* +* @retval 0 If success +* @retval -EIO IO error +* @retval -EINVAL Invalid argument +* @retval -EAGAIN IVSC device not ready +* @retval negative values for other errors +*/ +int ivsc_release_camera (void ); + +#else +static inline +int ivsc_acquire_camera (u32 nr_of_lanes,u64 link_freq, +void (*callback)(void *,enum ivsc_privacy_status ), +void *context) +{ +return 0 ; +} + +static inline int ivsc_release_camera (void ) +{ +return 0 ; +} + +#endif +#endif
Signature · 3 lines
Regards, Laurent Pinchart
RE: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc
Hi,
Thanks for your review
Quoted 32 lines
-----Original Message-----
From: Laurent Pinchart <[email protected]>
Sent: Wednesday, February 15, 2023 5:46 PMQuoted 27 lines · Hans de Goede
Hi Sakari,
Quoted 24 lines · Sakari Ailus
Hi Wentong,
Thanks for the patchset.
Quoted 9 lines · Wentong Wu
IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.
Quoted 5 lines
This commit defines the interfaces between IVSC and camera sensor driver in
include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image
Quoted 7 lines
pipeline.
Quoted 4 lines
Yes I happened to discuss this exact same thing with Laurent at FOSDEM and we also came to the conclusion that the IVSC chip should be modeled as a V4L2 sub-device.
Agreed.
Thanks for your quick review and conclusion, I'm fresh to media sub-system, is there any convention that I should follow to upstream this kind of v4l2 sub-device driver so that not too much back and forth?
Quoted 192 lines
Quoted 9 lines
Quoted 9 lines
I suppose the intended use cases assume a single instance of IVSC (as well as MEI) but there can, and often are, be multiple camera sensors in the system. The decision whether to request pass-through from IVCS can't be done in the camera sensor driver, and should not be visible to the camera sensor driver. Exposing IVSC as a V4L2 sub-device makes this trivial to address, as the IVSC driver's V4L2 sub-device video
s_stream()operation gets called before streaming is started.Quoted 3 lines
Quoted 3 lines
The information whether IVSC is found between the camera sensor and the host's CSI-2 receiver (IPU in this case) should come from system firmware and accessed most probably by what is called cio2-bridge at the
moment.
Quoted 103 lines
Quoted 103 lines
The privacy status can be a V4L2 control.
Also cc Hans.
Quoted 98 lines
Signed-off-by Wentong Wu <[email protected]>--- drivers/media/pci/intel/ivsc/Makefile | 1 + drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++++++++++++++++++++++++ include/linux/ivsc.h | 55 ++++++++++++++++++ 3 files changed, 140 insertions(+) create mode 100644
drivers/media/pci/intel/ivsc/ivsc.cdrivers/media/pci/intel/ivsc/Makefile +1
-0@@ -4,3 +4,4 @@ obj-$(CONFIG_INTEL_VSC) += mei_csi.o obj-$(CONFIG_INTEL_VSC) += mei_ace.o +obj-$(CONFIG_INTEL_VSC) += ivsc.odrivers/media/pci/intel/ivsc/ivsc.c new file +72
-0@@ -0,0 +1,84 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright (C) 2023 Intel Corporation. All rights reserved. +* Intel Visual Sensing Controller interface */ + +#include <linux/delay.h> +#include <linux/ivsc.h> +#include <linux/module.h> +#include <linux/mutex.h> + +#include "mei_ace.h" +#include "mei_csi.h" + +/* lock for ivsc APIs */ +static DEFINE_MUTEX (ivsc_mutex); + +int ivsc_acquire_camera (u32 nr_of_lanes,u64 link_freq, +void (*callback)(void *,enum ivsc_privacy_status ), +void *context) +{ +int ret; + +mutex_lock (&ivsc_mutex); + +/* switch camera sensor ownership to host */ + ret =ace_set_camera_owner (ACE_CAMERA_HOST ); +if (ret) + goto error; + +/* switch CSI-2 link to host */ + ret =csi_set_link_owner (CSI_LINK_HOST , callback, context); +if (ret) + goto release_camera; + +/* configure CSI-2 link */ + ret =csi_set_link_cfg (nr_of_lanes, link_freq); +if (ret) + goto release_csi; + +mutex_unlock (&ivsc_mutex); + +return 0 ; + +release_csi: +csi_set_link_owner (CSI_LINK_IVSC ,NULL ,NULL ); + +release_camera: +ace_set_camera_owner (ACE_CAMERA_IVSC ); + +error: +mutex_unlock (&ivsc_mutex); + +return ret; +} +EXPORT_SYMBOL_GPL (ivsc_acquire_camera); + +int ivsc_release_camera (void ) +{ +int ret; + +mutex_lock (&ivsc_mutex); + +/* switch CSI-2 link to IVSC */ + ret =csi_set_link_owner (CSI_LINK_IVSC ,NULL ,NULL ); +if (ret) + goto error; + +/* switch camera sensor ownership to IVSC */ + ret =ace_set_camera_owner (ACE_CAMERA_IVSC ); + +error: +mutex_unlock (&ivsc_mutex);Quoted 8 lines
Quoted 8 lines
drivers/media/pci/intel/ivsc/ivsc.c line 81
Quoted 8 lines
drivers/media/pci/intel/ivsc/ivsc.c new file +8
-0+ +return ret; +} +EXPORT_SYMBOL_GPL (ivsc_release_camera); + +MODULE_AUTHOR ("Wentong Wu <[email protected]>" ); +MODULE_AUTHOR ("Zhifeng Wang <[email protected]>" ); +MODULE_SOFTDEP ("pre: mei_csi mei_ace" );MODULE_DESCRIPTION("IVSC
Quoted 54 lines
Quoted 54 lines
Quoted 54 lines
drivers/media/pci/intel/ivsc/ivsc.c new file +1
-0+interface"); MODULE_LICENSE("GPL"); MODULE_IMPORT_NS(IVSC);include/linux/ivsc.h index renamed from include/linux/ivsc.h +0
-0
6572ca4f340c..bc9006cd6efc100644include/linux/ivsc.h +45
-0@@ -16,4 +16,59 @@ enum ivsc_privacy_status { IVSC_PRIVACY_MAX, }; +#if IS_ENABLED (CONFIG_INTEL_VSC ) +/* +* @brief Acquire camera sensor ownership to host and setup +* the CSI-2 link between host and IVSC +* +* IVSC provides a privacy mode. When the privacy mode is turned +* on, camera sensor can't be used. This means that both IVSC and +* host Image Processing Unit(IPU) can't get image data. And when +* this mode is turned on, host Image Processing Unit(IPU) driver +* is informed via the registered callback, so that user can be +* notified. +* +* @param nr_of_lanes Number of data lanes used on the CSI-2 link +* @param link_freq Frequency of the CSI-2 link +* @param callback The pointer of privacy callback function +* @param context Privacy callback function runtime context +* +* @retval 0 If success +* @retval -EIO IO error +* @retval -EINVAL Invalid argument +* @retval -EAGAIN IVSC device not ready +* @retval negative values for other errors */ int +ivsc_acquire_camera (u32 nr_of_lanes,u64 link_freq, +void (*callback)(void *,enum ivsc_privacy_status ), +void *context); + +/* +* @brief Release camera sensor ownership and stop the CSI-2 +* link between host and IVSC +* +* @retval 0 If success +* @retval -EIO IO error +* @retval -EINVAL Invalid argument +* @retval -EAGAIN IVSC device not ready +* @retval negative values for other errors */ int +ivsc_release_camera (void ); + +#else +static inline +int ivsc_acquire_camera (u32 nr_of_lanes,u64 link_freq, +void (*callback)(void *,enum ivsc_privacy_status ), +void *context) +{ +return 0 ; +}Quoted 8 lines
Quoted 8 lines
include/linux/ivsc.h enum ivsc_privacy_status { · line 74
Quoted 8 lines
include/linux/ivsc.h +7
-0+ +static inline int ivsc_release_camera (void ) { +return 0 ; +} + +#endif +#endif --
Regards,Laurent Pinchart
Re: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc
Hi Wentong,
Quoted 29 lines · Wentong
Quoted 28 lines · Laurent Pinchart
Quoted 26 lines · Hans de Goede
Quoted 16 lines · Sakari Ailus
Quoted 9 lines · Wentong Wu
IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.
Quoted 5 lines
This commit defines the interfaces between IVSC and camera sensor driver in
include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.Quoted 5 lines
The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image pipeline.
Yes I happened to discuss this exact same thing with Laurent at FOSDEM and we also came to the conclusion that the IVSC chip should be modeled as a V4L2 sub-device.
Quoted 6 lines
Quoted 1 line
Agreed.
Thanks for your quick review and conclusion, I'm fresh to media sub-system, is there any convention that I should follow to upstream this kind of v4l2 sub-device driver so that not too much back and forth?
This is a tentative proposal, as I'm not very familiar with the hardware architecture:
- The subdev should have two pads, a sink pad connected to the camera sensor, and a source pad connected to the CSI-2 receiver in the IPU.
- As for any new driver, the subdev driver should use the active state managed by the v4l2-subdev core. This requires calling v4l2_subdev_init_finalize() at probe time. See
https://git.linuxtv.org/media_tree.git/commit/?id=a2514b9a634a for an example of a subdev driver converted to this mechanism.
- As we're talking about CSI-2, the subdev driver should use the streams API that was recently merged, and in particular support the .get_frame_desc(), .set_routing(), .enable_streams() and .disable_streams() operations.
- I don't see a need to support V4L2 controls in the subdev driver, but I may be missing something.
- The driver should be validated with v4l2-compliance, part of v4l-utils.
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
I suppose the intended use cases assume a single instance of IVSC (as well as MEI) but there can, and often are, be multiple camera sensors in the system. The decision whether to request pass-through from IVCS can't be done in the camera sensor driver, and should not be visible to the camera sensor driver. Exposing IVSC as a V4L2 sub-device makes this trivial to address, as the IVSC driver's V4L2 sub-device video
s_stream()operation gets called before streaming is started.The information whether IVSC is found between the camera sensor and the host's CSI-2 receiver (IPU in this case) should come from system firmware and accessed most probably by what is called cio2-bridge at the moment.
The privacy status can be a V4L2 control.
Also cc Hans.
Quoted 7 lines
Signed-off-by Wentong Wu <[email protected]>--- drivers/media/pci/intel/ivsc/Makefile | 1 + drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++++++++++++++++++++++++ include/linux/ivsc.h | 55 ++++++++++++++++++ 3 files changed, 140 insertions(+) create mode 100644
drivers/media/pci/intel/ivsc/ivsc.c
[snip]
Signature · 3 lines
Regards, Laurent Pinchart
RE: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc
Quoted 63 lines
-----Original Message-----
From: Laurent Pinchart <[email protected]>
Sent: Friday, February 17, 2023 7:19 PMHi Wentong,
Quoted 30 lines · Wentong
Quoted 29 lines · Laurent Pinchart
Quoted 27 lines · Hans de Goede
Quoted 17 lines · Sakari Ailus
Quoted 10 lines · Wentong Wu
IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI
CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.Quoted 5 lines
This commit defines the interfaces between IVSC and camera sensor driver in
include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.Quoted 5 lines
The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image pipeline.
Yes I happened to discuss this exact same thing with Laurent at FOSDEM and we also came to the conclusion that the IVSC chip should be modeled as a V4L2 sub-device.
Quoted 6 lines
Quoted 1 line
Agreed.
Thanks for your quick review and conclusion, I'm fresh to media sub-system, is there any convention that I should follow to upstream this kind of v4l2 sub-device driver so that not too much back and forth?
This is a tentative proposal, as I'm not very familiar with the hardware architecture:
- The subdev should have two pads, a sink pad connected to the camera sensor, and a source pad connected to the CSI-2 receiver in the IPU.
- As for any new driver, the subdev driver should use the active state managed by the v4l2-subdev core. This requires calling
v4l2_subdev_init_finalize()at probe time. See
https://git.linuxtv.org/media_tree.git/commit/?id=a2514b9a634a for an example of a subdev driver converted to this mechanism.- As we're talking about CSI-2, the subdev driver should use the streams API that was recently merged, and in particular support the .get_frame_desc(), .set_routing(), .enable_streams() and .disable_streams() operations.
Quoted 6 lines
- I don't see a need to support V4L2 controls in the subdev driver, but I may be missing something.
- The driver should be validated with v4l2-compliance, part of v4l-utils.
Thanks for the detail, but I have one more question, during probe of sensor(v4l2-sudev) driver, it will configure camera sensor connected to IVSC via I2C, but before that it should acquire camera sensor's ownership from IVSC, how v4l2 framework guarantee this?
Thanks,
Wentong
Quoted 33 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
I suppose the intended use cases assume a single instance of IVSC (as well as MEI) but there can, and often are, be multiple camera sensors in the system. The decision whether to request pass-through from IVCS can't be done in the camera sensor driver, and should not be visible to the camera sensor driver. Exposing IVSC as a V4L2 sub-device makes this trivial to address, as the IVSC driver's V4L2 sub-device video
s_stream()operationgets called before streaming is started.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
The information whether IVSC is found between the camera sensor and the host's CSI-2 receiver (IPU in this case) should come from system firmware and accessed most probably by what is called cio2-bridge at the moment.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
The privacy status can be a V4L2 control.
Also cc Hans.
Quoted 4 lines
Signed-off-by Wentong Wu <[email protected]>--- drivers/media/pci/intel/ivsc/Makefile | 1 + drivers/media/pci/intel/ivsc/ivsc.c | 84+++++++++++++++++++++++++++
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
include/linux/ivsc.h | 55 ++++++++++++++++++ 3 files changed, 140 insertions(+) create mode 100644
drivers/media/pci/intel/ivsc/ivsc.c[snip]
--
Regards,Laurent Pinchart
Sakari Ailus
maintainerRe: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc
Hi Wentong,
Quoted 66 lines · Wentong
Quoted 66 lines
-----Original Message-----
From: Laurent Pinchart <[email protected]>
Sent: Friday, February 17, 2023 7:19 PMHi Wentong,
Quoted 30 lines · Wentong
Quoted 29 lines · Laurent Pinchart
Quoted 27 lines · Hans de Goede
Quoted 17 lines · Sakari Ailus
Quoted 10 lines · Wentong Wu
IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI
CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.Quoted 5 lines
This commit defines the interfaces between IVSC and camera sensor driver in
include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.Quoted 5 lines
The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image pipeline.
Yes I happened to discuss this exact same thing with Laurent at FOSDEM and we also came to the conclusion that the IVSC chip should be modeled as a V4L2 sub-device.
Quoted 6 lines
Quoted 1 line
Agreed.
Thanks for your quick review and conclusion, I'm fresh to media sub-system, is there any convention that I should follow to upstream this kind of v4l2 sub-device driver so that not too much back and forth?
This is a tentative proposal, as I'm not very familiar with the hardware architecture:
- The subdev should have two pads, a sink pad connected to the camera sensor, and a source pad connected to the CSI-2 receiver in the IPU.
- As for any new driver, the subdev driver should use the active state managed by the v4l2-subdev core. This requires calling
v4l2_subdev_init_finalize()at probe time. See
https://git.linuxtv.org/media_tree.git/commit/?id=a2514b9a634a for an example of a subdev driver converted to this mechanism.- As we're talking about CSI-2, the subdev driver should use the streams API that was recently merged, and in particular support the .get_frame_desc(), .set_routing(), .enable_streams() and .disable_streams() operations.
- I don't see a need to support V4L2 controls in the subdev driver, but I may be missing something.
Quoted 8 lines
Quoted 3 lines
- The driver should be validated with v4l2-compliance, part of v4l-utils.
Thanks for the detail, but I have one more question, during probe of sensor(v4l2-sudev) driver, it will configure camera sensor connected to IVSC via I2C, but before that it should acquire camera sensor's ownership from IVSC, how v4l2 framework guarantee this?
Please wrap the lines at 74 characters or so when replying.
Do you mean accessing the sensor via I²C also requires acquiring the sensor from IVSC?
Signature · 3 lines
Regards, Sakari Ailus
RE: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc
Quoted 81 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Monday, February 20, 2023 4:37 PMHi Wentong,
Quoted 10 lines · Wentong
Quoted 10 lines
-----Original Message-----
From: Laurent Pinchart <[email protected]>
Sent: Friday, February 17, 2023 7:19 PMHi Wentong,
Quoted 1 line · Wentong
On Sent: Wednesday, February 15, 2023 5:46 PM, Laurent Pinchart wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines · Hans de Goede
Quoted 7 lines · Sakari Ailus
Quoted 6 lines · Wentong Wu
Quoted 5 lines
IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI
CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results toISH.
Quoted 53 lines
Quoted 52 lines
Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
Quoted 11 lines
Quoted 10 lines
On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.
This commit defines the interfaces between IVSC and camera sensor driver in
include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.Quoted 5 lines
The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image pipeline.
Yes I happened to discuss this exact same thing with Laurent at FOSDEM and we also came to the conclusion that the IVSC chip should be modeled as a V4L2 sub-device.
Quoted 6 lines
Quoted 1 line
Agreed.
Thanks for your quick review and conclusion, I'm fresh to media sub-system, is there any convention that I should follow to upstream this kind of v4l2 sub-device driver so that not too much back and forth?
This is a tentative proposal, as I'm not very familiar with the hardware
architecture:- The subdev should have two pads, a sink pad connected to the camera sensor, and a source pad connected to the CSI-2 receiver in the IPU.
- As for any new driver, the subdev driver should use the active state managed by the v4l2-subdev core. This requires calling
v4l2_subdev_init_finalize()at probe time. See
https://git.linuxtv.org/media_tree.git/commit/?id=a2514b9a634a for an example of a subdev driver converted to this mechanism.- As we're talking about CSI-2, the subdev driver should use the streams API that was recently merged, and in particular support the .get_frame_desc(), .set_routing(), .enable_streams() and .disable_streams() operations.
- I don't see a need to support V4L2 controls in the subdev driver, but I may be missing something.
- The driver should be validated with v4l2-compliance, part of v4l-utils.
Quoted 6 lines
Quoted 4 lines
Thanks for the detail, but I have one more question, during probe of sensor(v4l2-sudev) driver, it will configure camera sensor connected to IVSC via I2C, but before that it should acquire camera sensor's ownership from IVSC, how v4l2 framework guarantee this?
Please wrap the lines at 74 characters or so when replying.
Thanks, I will.
Quoted 3 lines
Do you mean accessing the sensor via I²C also requires acquiring the sensor from IVSC?
Yes
Quoted 5 lines
--
Regards,Sakari Ailus
Re: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc
Hi,
Quoted 88 lines · Wentong
Quoted 81 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Monday, February 20, 2023 4:37 PMHi Wentong,
Quoted 10 lines · Wentong
Quoted 10 lines
-----Original Message-----
From: Laurent Pinchart <[email protected]>
Sent: Friday, February 17, 2023 7:19 PMHi Wentong,
Quoted 1 line · Wentong
On Sent: Wednesday, February 15, 2023 5:46 PM, Laurent Pinchart wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines · Hans de Goede
Quoted 7 lines · Sakari Ailus
Quoted 6 lines · Wentong Wu
Quoted 5 lines
IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI
CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results toISH.
Quoted 53 lines
Quoted 52 lines
Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
Quoted 11 lines
Quoted 10 lines
On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.
This commit defines the interfaces between IVSC and camera sensor driver in
include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.Quoted 5 lines
The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image pipeline.
Yes I happened to discuss this exact same thing with Laurent at FOSDEM and we also came to the conclusion that the IVSC chip should be modeled as a V4L2 sub-device.
Quoted 6 lines
Quoted 1 line
Agreed.
Thanks for your quick review and conclusion, I'm fresh to media sub-system, is there any convention that I should follow to upstream this kind of v4l2 sub-device driver so that not too much back and forth?
This is a tentative proposal, as I'm not very familiar with the hardware
architecture:- The subdev should have two pads, a sink pad connected to the camera sensor, and a source pad connected to the CSI-2 receiver in the IPU.
- As for any new driver, the subdev driver should use the active state managed by the v4l2-subdev core. This requires calling
v4l2_subdev_init_finalize()at probe time. See
https://git.linuxtv.org/media_tree.git/commit/?id=a2514b9a634a for an example of a subdev driver converted to this mechanism.- As we're talking about CSI-2, the subdev driver should use the streams API that was recently merged, and in particular support the .get_frame_desc(), .set_routing(), .enable_streams() and .disable_streams() operations.
- I don't see a need to support V4L2 controls in the subdev driver, but I may be missing something.
- The driver should be validated with v4l2-compliance, part of v4l-utils.
Quoted 6 lines
Quoted 4 lines
Thanks for the detail, but I have one more question, during probe of sensor(v4l2-sudev) driver, it will configure camera sensor connected to IVSC via I2C, but before that it should acquire camera sensor's ownership from IVSC, how v4l2 framework guarantee this?
Please wrap the lines at 74 characters or so when replying.
Quoted 7 lines
Thanks, I will.
Quoted 3 lines
Do you mean accessing the sensor via I²C also requires acquiring the sensor from IVSC?
Yes
Hmm, that is going to be a problem since we really want to have independent driver for the 2 which are not aware of each other.
I think that maybe we can model this part of the ivsc functionality as an i2c-mux. But then we will need to somehow change the parent of the i2c device for the sensor to the output of this mux ...
I guess this means adding some code (some hack likely) to drivers/platform/x86/intel/int3472/discrete.c which' proe function is already guaranteed to run before the sensor's i2c-client gets instantiated, because of the ACPI _DEP on the INT3472 ACPI device in the DSDT.
This is going to be a tricky problem to tackle.
Regards,
Hans
Re: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc
Quoted 98 lines · Hans de Goede
Quoted 81 lines · Wentong
Quoted 81 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Monday, February 20, 2023 4:37 PMHi Wentong,
Quoted 73 lines · Wentong
Quoted 62 lines
-----Original Message-----
From: Laurent Pinchart <[email protected]>
Sent: Friday, February 17, 2023 7:19 PMHi Wentong,
Quoted 30 lines · Wentong
Quoted 29 lines · Laurent Pinchart
Quoted 27 lines · Hans de Goede
Quoted 17 lines · Sakari Ailus
Quoted 10 lines · Wentong Wu
IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI
CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI
CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.Quoted 5 lines
This commit defines the interfaces between IVSC and camera sensor driver in
include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.Quoted 5 lines
The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image pipeline.
Yes I happened to discuss this exact same thing with Laurent at FOSDEM and we also came to the conclusion that the IVSC chip should be modeled as a V4L2 sub-device.
Quoted 6 lines
Quoted 1 line
Agreed.
Thanks for your quick review and conclusion, I'm fresh to media sub-system, is there any convention that I should follow to upstream this kind of v4l2 sub-device driver so that not too much back and forth?
This is a tentative proposal, as I'm not very familiar with the hardware
architecture:- The subdev should have two pads, a sink pad connected to the camera sensor, and a source pad connected to the CSI-2 receiver in the IPU.
- As for any new driver, the subdev driver should use the active state managed by the v4l2-subdev core. This requires calling
v4l2_subdev_init_finalize()at probe time. See
https://git.linuxtv.org/media_tree.git/commit/?id=a2514b9a634a for an example of a subdev driver converted to this mechanism.- As we're talking about CSI-2, the subdev driver should use the streams API that was recently merged, and in particular support the .get_frame_desc(), .set_routing(), .enable_streams() and .disable_streams() operations.
Quoted 6 lines
- I don't see a need to support V4L2 controls in the subdev driver, but I may be missing something.
- The driver should be validated with v4l2-compliance, part of v4l-utils.
Thanks for the detail, but I have one more question, during probe of sensor(v4l2-sudev) driver, it will configure camera sensor connected to IVSC via I2C, but before that it should acquire camera sensor's ownership from IVSC, how v4l2 framework guarantee this?
Quoted 8 lines
Quoted 1 line
Please wrap the lines at 74 characters or so when replying.
Thanks, I will.
Quoted 2 lines
Do you mean accessing the sensor via I²C also requires acquiring the sensor from IVSC?
Yes
Hmm, that is going to be a problem since we really want to have independent driver for the 2 which are not aware of each other.
I think that maybe we can model this part of the ivsc functionality as an i2c-mux. But then we will need to somehow change the parent of the i2c device for the sensor to the output of this mux ...
Quoted 7 lines
I guess this means adding some code (some hack likely) to
drivers/platform/x86/intel/int3472/discrete.cwhich' proe function is already guaranteed to run before the sensor's i2c-client gets instantiated, because of the ACPI_DEPon the INT3472 ACPI device in the DSDT.This is going to be a tricky problem to tackle.
From a framework point of view I would also recommend i2c-mux, but when it comes to the ACPI integration... I can feel your pain and I'm sorry about that :-(
Signature · 3 lines
Regards, Laurent Pinchart
RE: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc
In reply to Laurent PinchartHi Laurent,
Quoted 61 lines
-----Original Message-----
From: Laurent Pinchart <[email protected]>
Sent: Friday, February 17, 2023 7:19 PMHi Wentong,
Quoted 30 lines · Wentong
Quoted 29 lines · Laurent Pinchart
Quoted 27 lines · Hans de Goede
Quoted 17 lines · Sakari Ailus
Quoted 10 lines · Wentong Wu
IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI
CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.Quoted 5 lines
This commit defines the interfaces between IVSC and camera sensor driver in
include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.Quoted 5 lines
The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image pipeline.
Yes I happened to discuss this exact same thing with Laurent at FOSDEM and we also came to the conclusion that the IVSC chip should be modeled as a V4L2 sub-device.
Quoted 6 lines
Quoted 1 line
Agreed.
Thanks for your quick review and conclusion, I'm fresh to media sub-system, is there any convention that I should follow to upstream this kind of v4l2 sub-device driver so that not too much back and forth?
This is a tentative proposal, as I'm not very familiar with the hardware architecture:
- The subdev should have two pads, a sink pad connected to the camera sensor, and a source pad connected to the CSI-2 receiver in the IPU.
- As for any new driver, the subdev driver should use the active state managed by the v4l2-subdev core. This requires calling
v4l2_subdev_init_finalize()at probe time. See
https://git.linuxtv.org/media_tree.git/commit/?id=a2514b9a634a for an example of a subdev driver converted to this mechanism.- As we're talking about CSI-2, the subdev driver should use the streams API that was recently merged, and in particular support the .get_frame_desc(), .set_routing(), .enable_streams() and .disable_streams() operations.
Quoted 5 lines
- I don't see a need to support V4L2 controls in the subdev driver, but I may be missing something.
- The driver should be validated with v4l2-compliance, part of v4l-utils.
Thanks, one more question, because ivsc just routes the received data from sensor and no format convert, how we handle the get_fmt and set_fmt callback for ivsc?
Thanks
Wentong
Quoted 34 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
I suppose the intended use cases assume a single instance of IVSC (as well as MEI) but there can, and often are, be multiple camera sensors in the system. The decision whether to request pass-through from IVCS can't be done in the camera sensor driver, and should not be visible to the camera sensor driver. Exposing IVSC as a V4L2 sub-device makes this trivial to address, as the IVSC driver's V4L2 sub-device video
s_stream()operationgets called before streaming is started.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
The information whether IVSC is found between the camera sensor and the host's CSI-2 receiver (IPU in this case) should come from system firmware and accessed most probably by what is called cio2-bridge at the moment.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
The privacy status can be a V4L2 control.
Also cc Hans.
Quoted 4 lines
Signed-off-by Wentong Wu <[email protected]>--- drivers/media/pci/intel/ivsc/Makefile | 1 + drivers/media/pci/intel/ivsc/ivsc.c | 84+++++++++++++++++++++++++++
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
include/linux/ivsc.h | 55 ++++++++++++++++++ 3 files changed, 140 insertions(+) create mode 100644
drivers/media/pci/intel/ivsc/ivsc.c[snip]
--
Regards,Laurent Pinchart
RE: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc
In reply to Sakari AilusHi Sakari,
few questions as switching to v4l2 sub-dev framework.
Quoted 35 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Wednesday, February 15, 2023 12:06 AMHi Wentong,
Thanks for the patchset.
Quoted 8 lines · Wentong Wu
IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.
Quoted 5 lines
This commit defines the interfaces between IVSC and camera sensor driver in
include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image pipeline.
I suppose the intended use cases assume a single instance of IVSC (as well as MEI) but there can, and often are, be multiple camera sensors in the system. The decision whether to request pass-through from IVCS can't be done in the camera sensor driver, and should not be visible to the camera sensor driver. Exposing IVSC as a V4L2 sub-device makes this trivial to address, as the IVSC driver's V4L2 sub-device video
s_stream()operation gets called before streaming is started.
Quoted 5 lines
The information whether IVSC is found between the camera sensor and the host's CSI-2 receiver (IPU in this case) should come from system firmware and accessed most probably by what is called cio2-bridge at the moment.
The privacy status can be a V4L2 control.
This should be a control or event? If control, how user-space handle privacy stuff?
For the required link freq and lane number, is v4l2 control the correct way to configure them? If yes, seems there is no CID value for them so that we should custom some CID value(link freqm, lane number, and privacy) for ivsc in linux/v4l2-controls.h, is this acceptable?
Thanks,
Wentong
Quoted 4 lines
Also cc Hans.
Quoted 1 line
Sakari Ailus
maintainerRe: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc
Hi Wentong,
Quoted 39 lines · Wentong
Hi Sakari,
few questions as switching to v4l2 sub-dev framework.
Quoted 35 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Wednesday, February 15, 2023 12:06 AMHi Wentong,
Thanks for the patchset.
Quoted 8 lines · Wentong Wu
IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are defined to sync the ownership.
Quoted 5 lines
This commit defines the interfaces between IVSC and camera sensor driver in
include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image pipeline.
I suppose the intended use cases assume a single instance of IVSC (as well as MEI) but there can, and often are, be multiple camera sensors in the system. The decision whether to request pass-through from IVCS can't be done in the camera sensor driver, and should not be visible to the camera sensor driver. Exposing IVSC as a V4L2 sub-device makes this trivial to address, as the IVSC driver's V4L2 sub-device video
s_stream()operation gets called before streaming is started.
Quoted 8 lines
Quoted 5 lines
The information whether IVSC is found between the camera sensor and the host's CSI-2 receiver (IPU in this case) should come from system firmware and accessed most probably by what is called cio2-bridge at the moment.
The privacy status can be a V4L2 control.
This should be a control or event? If control, how user-space handle privacy stuff?
Changing control events generates events for the user space.
<URL:https://hverkuil.home.xs4all.nl/spec/userspace-api/v4l/dev-event.html>
Quoted 5 lines
For the required link freq and lane number, is v4l2 control the correct way to configure them? If yes, seems there is no CID value for them so that we should custom some CID
value(link freqm, lane number, and privacy)for ivsc inlinux/v4l2-controls.h, is this acceptable?
You should obtain these using the V4L2 fwnode interface. Please see e.g. drivers/media/platform/renesas/rzg2l-cru/rzg2l-csi2.c how that driver registers a V4L2 async sub-device and a V4L2 async notifier.
Signature · 3 lines
Kind regards, Sakari Ailus
RE: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc
Quoted 57 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Tuesday, February 28, 2023 4:24 PMHi Wentong,
Quoted 20 lines · Wentong
Hi Sakari,
few questions as switching to v4l2 sub-dev framework.
Quoted 16 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Wednesday, February 15, 2023 12:06 AMHi Wentong,
Thanks for the patchset.
Quoted 7 lines · Wentong Wu
IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are
defined to sync the ownership.
Quoted 11 lines
Quoted 11 lines
Quoted 6 lines
This commit defines the interfaces between IVSC and camera sensor driver in
include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image
pipeline.
Quoted 8 lines
Quoted 8 lines
I suppose the intended use cases assume a single instance of IVSC (as well as
MEI) but there can, and often are, be multiple camera sensors in the system. The decision whether to request pass-through from IVCS can't be done in the camera sensor driver, and should not be visible to the camera sensor driver. Exposing IVSC as a V4L2 sub-device makes this trivial to address, as the IVSC driver's V4L2 sub-device videos_stream()operation gets called before streaming is started.
Quoted 4 lines
Quoted 4 lines
The information whether IVSC is found between the camera sensor and the host's CSI-2 receiver (IPU in this case) should come from system firmware and accessed most probably by what is called cio2-bridge at the
moment.
Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
The privacy status can be a V4L2 control.
This should be a control or event? If control, how user-space handle privacy stuff?
Changing control events generates events for the user space.
<URL:https://hverkuil.home.xs4all.nl/spec/userspace-api/v4l/dev-event.html>
Ok, V4L2_EVENT_CTRL event which reports 'struct v4l2_event_ctrl' data to user space
Quoted 11 lines
Quoted 7 lines
For the required link freq and lane number, is v4l2 control the correct way to configure them? If yes, seems there is no CID value for them so that we should custom some CID value(link freqm, lane number, and
privacy) for ivsc inlinux/v4l2-controls.h, is this acceptable?You should obtain these using the V4L2 fwnode interface. Please see e.g.
drivers/media/platform/renesas/rzg2l-cru/rzg2l-csi2.chow that driver registers a V4L2 async sub-device and a V4L2 async notifier.
Ok, something like v4l2_ctrl_find(source->ctrl_handler, V4L2_CID_LINK_FREQ); is to get sensor's link frequency, and the code like 'v4l2_subdev_call(source, pad, get_mbus_config, source, &mbus_config); num_of_lanes = mbus_config.bus.mipi_csi2.num_data_lanes;' is to get sensor's lane number.
BR,
Wentong
Quoted 5 lines
--
Kind regards,Sakari Ailus
Sakari Ailus
maintainerRe: [PATCH v2 3/3] media: pci: intel: ivsc: Add acquire/release API for ivsc
Hi Wentong,
Quoted 80 lines · Wentong
Quoted 57 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Tuesday, February 28, 2023 4:24 PMHi Wentong,
Quoted 20 lines · Wentong
Hi Sakari,
few questions as switching to v4l2 sub-dev framework.
Quoted 16 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Wednesday, February 15, 2023 12:06 AMHi Wentong,
Thanks for the patchset.
Quoted 7 lines · Wentong Wu
IVSC directly connects to camera sensor on source side, and on output side it not only connects ISH via I2C, but also exposes MIPI CSI-2 interface to output camera sensor data. IVSC can use the camera sensor data to do AI algorithm, and send the results to ISH. On the other end, IVSC can share camera sensor to host by routing the raw camera sensor data to the exposed MIPI CSI-2 interface. But they can not work at the same time, so software APIs are
defined to sync the ownership.
Quoted 11 lines
Quoted 11 lines
Quoted 6 lines
This commit defines the interfaces between IVSC and camera sensor driver in
include/linux/ivsc.h. The camera driver controls ownership of the CSI-2 link and sensor with the acquire/release APIs. When acquiring camera, lane number and link freq are also required by IVSC frame router.The more I learn about this system, the more I'm inclined to think this functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do anything to the data (as long as it directs it towards the CSI-2 receiver in the SoC), but it is definitely part of the image
pipeline.
Quoted 8 lines
Quoted 8 lines
I suppose the intended use cases assume a single instance of IVSC (as well as
MEI) but there can, and often are, be multiple camera sensors in the system. The decision whether to request pass-through from IVCS can't be done in the camera sensor driver, and should not be visible to the camera sensor driver. Exposing IVSC as a V4L2 sub-device makes this trivial to address, as the IVSC driver's V4L2 sub-device videos_stream()operation gets called before streaming is started.
Quoted 4 lines
Quoted 4 lines
The information whether IVSC is found between the camera sensor and the host's CSI-2 receiver (IPU in this case) should come from system firmware and accessed most probably by what is called cio2-bridge at the
moment.
Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
The privacy status can be a V4L2 control.
This should be a control or event? If control, how user-space handle privacy stuff?
Changing control events generates events for the user space.
<URL:https://hverkuil.home.xs4all.nl/spec/userspace-api/v4l/dev-event.html>
Ok,
V4L2_EVENT_CTRLevent which reports 'structv4l2_event_ctrl' data to user spaceQuoted 11 lines
Quoted 7 lines
For the required link freq and lane number, is v4l2 control the correct way to configure them? If yes, seems there is no CID value for them so that we should custom some CID value(link freqm, lane number, and
privacy) for ivsc inlinux/v4l2-controls.h, is this acceptable?You should obtain these using the V4L2 fwnode interface. Please see e.g.
drivers/media/platform/renesas/rzg2l-cru/rzg2l-csi2.chow that driver registers a V4L2 async sub-device and a V4L2 async notifier.
Quoted 5 lines
Ok, something like
v4l2_ctrl_find(source->ctrl_handler, V4L2_CID_LINK_FREQ); is to get sensor's link frequency, and the code like 'v4l2_subdev_call(source, pad, get_mbus_config, source, &mbus_config);num_of_lanes= mbus_config.bus.mipi_csi2.num_data_lanes;' is to get sensor's lane number.
Please use the V4L2 fwnode interface instead.
The privacy control should only be needed in the user space.
Signature · 3 lines
Regards, Sakari Ailus
Quoted 1 line · Wentong Wu
v2 without change log, why?
Quoted 169 lines
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a
companion chip designed to provide secure and low power vision capability
to IA platforms. IVSC is available in existing commercial platforms from
multiple OEMs.The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link
and processes the image data with the embedded AI engine. The detected
events are sent over I2C to ISH (Intel Sensor Hub) for additional data
fusion from multiple sensors. The fusion results are used to implement
advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up systemSince the Image Processing Unit(IPU) used on the host processor needs to
configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and
IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU
driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel
MEI (The Intel Management Engine) commands. These two hardware modules
have two different MEI UUIDs to enumerate. These hardware modules are:
- ACE (Algorithm Context Engine): This module is for algorithm computing
when IVSC owns camera sensor. Also ACE module controls camera sensor's
ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route camera sensor data either to IVSC or to host for IPU driver and application.IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't
get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
Implementation:
There are two different drivers to handle ACE and CSI hardware modules
inside IVSC.
-mei_csi: MEI client driver to send commands and receive notifications
from CSI module.
-mei_ace: MEI client driver to send commands and get status from ACE
module.
Interface is exposed viaivsc.hto acquire and release camera sensor and
CSI-2 link.Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
---------------------------------------------------------------------
--------Quoted 21 lines
Host
Processor
|
|
----------------- ----------------- ------------- -- || | | | | | I2C | | IPU | | ISH | |camera driver|--| | | | | | | | | | ----------------- ----------------- ------------- -- | | | | | | | | | ------------- -- | | | | | | | |Quoted 8 lines
| | | IVSC driver | | | | | | | | | | | ------------- -- | | | | | | |----------------|-----------------------|----------------------|----- ----|---
| CSI | I2C |SPI | | | | | ----------------|-----------------------|---------------- | | > IVSC | | > | | > | | > | | > ----------------- ----------------- | > | | > | | | | | > | | > | CSI | | ACE | |------ > | | > | | | | > | |Quoted 4 lines
----------------- -----------------
| | | | I2C | |----------------|-----------------------|---------------- |
| CSI| | | | | -------------------------------- | | | I2C | | camera sensor |------------------------- ----| | |--------------------------------
Wentong Wu (3):
media: pci: intel: ivsc: Add CSI submodule
media: pci: intel: ivsc: Add ACE submodule
media: pci: intel: ivsc: Add acquire/release API for ivscdrivers/media/pci/Kconfig | 1 + drivers/media/pci/intel/Makefile | 2 + drivers/media/pci/intel/ivsc/Kconfig | 12 + drivers/media/pci/intel/ivsc/Makefile | 7 + drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++
drivers/media/pci/intel/ivsc/mei_ace.c| 472
+++++++++++++++++++++++++drivers/media/pci/intel/ivsc/mei_ace.h | 36 ++ drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++ drivers/media/pci/intel/ivsc/mei_csi.h | 60 ++++ include/linux/ivsc.h | 74 ++++ 10 files changed, 1090 insertions(+)create mode 100644
drivers/media/pci/intel/ivsc/Kconfigcreate mode 100644drivers/media/pci/intel/ivsc/Makefilecreate mode 100644drivers/media/pci/intel/ivsc/ivsc.ccreate mode 100644drivers/media/pci/intel/ivsc/mei_ace.ccreate mode 100644drivers/media/pci/intel/ivsc/mei_ace.hcreate mode 100644drivers/media/pci/intel/ivsc/mei_csi.ccreate mode 100644drivers/media/pci/intel/ivsc/mei_csi.hcreate mode 100644include/linux/ivsc.h
Hi Srinivas,
Thanks for your review
Quoted 8 lines
-----Original Message-----
From: Pandruvada, Srinivas <[email protected]>
Sent: Monday, February 13, 2023 11:21 AMQuoted 1 line · Wentong Wu
v2 without change log, why?
This addressed 'mei_csi.c:(.text+0x1a6): undefined reference to __udivdi3' issue on i386 arch. I will add change log on following version.
Quoted 153 lines
Quoted 153 lines
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up systemSince the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware modules
are:
- ACE (Algorithm Context Engine): This module is for algorithm computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route camera sensor data either to IVSC or to host for IPU driver and application.IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.
In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
Implementation:
There are two different drivers to handle ACE and CSI hardware modules inside IVSC.
-mei_csi: MEI client driver to send commands and receive notifications from CSI module.
-mei_ace: MEI client driver to send commands and get status from ACE module.
Interface is exposed viaivsc.hto acquire and release camera sensor and
CSI-2 link.Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
---------------------------------------------------------------------
--------Quoted 21 lines
Host
Processor
||
----------------- ----------------- ------------- -- || | | | | | I2C | | IPU | | ISH | |camera driver|--| | | | | | | | | | ----------------- ----------------- ------------- -- | | | | | | | | | ------------- -- | | | | | | | |Quoted 8 lines
| | | IVSC driver | | | | | | | | | | | ------------- -- | | | | | | |----------------|-----------------------|----------------------|----- ----|---
| CSI | I2C |SPI | | | | | ----------------|-----------------------|---------------- | | > IVSC | | > | | > | | > | | > ----------------- ----------------- | > | | > | | | | | > | | > | CSI | | ACE | |------ > | | > | | | | > | |Quoted 4 lines
----------------- -----------------
| | | | I2C | |----------------|-----------------------|---------------- |
| CSI| | | | | -------------------------------- | | | I2C | | camera sensor |------------------------- ----| | |--------------------------------
Wentong Wu (3):
media: pci: intel: ivsc: Add CSI submodule
media: pci: intel: ivsc: Add ACE submodule
media: pci: intel: ivsc: Add acquire/release API for ivscdrivers/media/pci/Kconfig | 1 + drivers/media/pci/intel/Makefile | 2 + drivers/media/pci/intel/ivsc/Kconfig | 12 + drivers/media/pci/intel/ivsc/Makefile | 7 + drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++
drivers/media/pci/intel/ivsc/mei_ace.c| 472
+++++++++++++++++++++++++drivers/media/pci/intel/ivsc/mei_ace.h | 36 ++ drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++ drivers/media/pci/intel/ivsc/mei_csi.h | 60 ++++ include/linux/ivsc.h | 74 ++++ 10 files changed, 1090 insertions(+)create mode 100644
drivers/media/pci/intel/ivsc/Kconfigcreate mode 100644drivers/media/pci/intel/ivsc/Makefilecreate mode 100644drivers/media/pci/intel/ivsc/ivsc.ccreate mode 100644drivers/media/pci/intel/ivsc/mei_ace.ccreate mode 100644drivers/media/pci/intel/ivsc/mei_ace.hcreate mode 100644drivers/media/pci/intel/ivsc/mei_csi.ccreate mode 100644drivers/media/pci/intel/ivsc/mei_csi.hcreate mode 100644include/linux/ivsc.h
A message that is not in the archive (sent off-list or not yet received): [email protected]
RE: [PATCH v2 1/3] media: pci: intel: ivsc: Add CSI submodule
Hi Hillf Danton,
Thanks for your review
Quoted 5 lines
-----Original Message-----
From: Hillf Danton <[email protected]>
Sent: Monday, February 13, 2023 11:42 AMOn Mon, 13 Feb 2023 10:23:45 +0800 Wentong Wu <[email protected]>
Quoted 9 lines
Review
Quoted 7 lines
(quoted diff) +7
-0+ +/* send a command to firmware and mutex must be held by caller */ +static int mei_csi_send(u8 *buf, size_t len) { + struct csi_cmd *cmd = (struct csi_cmd *)buf; + int ret; + + reinit_completion(&csi->cmd_completion);Could you specify why reinit is needed here?
This allows new command(e.g. 'get status' command to check current firmware status) to be downloaded to firmware for debugging purpose in case no response for current command though I have never saw this happen.
Quoted 1 line
What is hurt without it?
No hurt for current implementation. I can remove it.
Quoted 2 lines
Same question for the reinit in 2/3.
same ack for 2/3
Quoted 14 lines
Quoted 14 lines
(quoted diff) +13
-0+ + ret = mei_cldev_send(csi->cldev, buf, len); + if (ret < 0) + goto out; + + ret = wait_for_completion_killable_timeout(&csi->cmd_completion, + CSI_CMD_TIMEOUT); + if (ret < 0) { + goto out; + } else if (!ret) { + ret = -ETIMEDOUT; + goto out; + }
A message that is not in the archive (sent off-list or not yet received): [email protected]
RE: [PATCH v2 1/3] media: pci: intel: ivsc: Add CSI submodule
Quoted 13 lines
-----Original Message-----
From: Hillf Danton <[email protected]>
Sent: Tuesday, February 14, 2023 4:36 PMOn Tue, 14 Feb 2023 06:25:35 +0000 Wentong Wu <[email protected]>
Quoted 6 lines
This allows new command(e.g. 'get status' command to check current firmware=
status) to be downloaded to firmware for debugging purpose in case no
resp= onse for current command though I have never saw this happen. =20
Quoted 4 lines
Thanks for your specification.
Another question, is it likely for the wakeup to come after reinit because of for instance timeout on the waiter side?
Thanks, first I want to clarify why we have timeout here, in case firmware stuck somehow or IVSC device is removed suddenly, calling thread can't be wakeup if wait forever here.
And firmware gets high priority to handle command and response, also there is timeout mechanism inside firmware, I think there will be some problem with firmware if driver gets command response after 5s which will be captured by driver code(if (csi->cmd_response.cmd_id != cmd->cmd_id)) but I have never saw this happen.
BR,
Wentong
A message that is not in the archive (sent off-list or not yet received): [email protected]
RE: [PATCH v2 1/3] media: pci: intel: ivsc: Add CSI submodule
Quoted 39 lines
-----Original Message-----
From: Hillf Danton <[email protected]>
Sent: Wednesday, February 15, 2023 7:54 AMOn Tue, 14 Feb 2023 12:40:12 +0000 Wentong Wu <[email protected]>
Quoted 10 lines
Quoted 10 lines
-----Original Message-----
From: Hillf Danton <[email protected]>
Sent: Tuesday, February 14, 2023 4:36 PM
=20
On Tue, 14 Feb 2023 06:25:35 +0000 Wentong Wu <[email protected]>Quoted 5 lines
This allows new command(e.g. 'get status' command to check current firmware=3D
status) to be downloaded to firmware for debugging purpose in
case no resp=3D onse for current command though I have never saw thishappen.
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
=3D20
=20
Thanks for your specification.
=20
Another question, is it likely for the wakeup to come after reinit becaus=Quoted 6 lines
e of for
Quoted 1 line
instance timeout on the waiter side?
Thanks, first I want to clarify why we have timeout here, in case firmware = stuck somehow or IVSC device is removed suddenly, calling thread can't be w= akeup if wait forever here.
Fair enough on the waiter side.
Quoted 6 lines
And firmware gets high priority to handle command and response, also there = is timeout mechanism inside firmware, I think there will be some problem wi= th firmware if driver gets command response after 5s which will be captured= by driver
code(if (csi->cmd_response.cmd_id !=3D cmd->cmd_id))but I have = never saw this happen.
Quoted 3 lines
The problem with firmware could be emulated by directly skipping the wait for completion and returning ETIMEDOUT, in order to see what will happen if wakeup comes later than timeout.
Thanks, but what do you mean "emulated"?
A message that is not in the archive (sent off-list or not yet received): [email protected]
RE: [PATCH v2 1/3] media: pci: intel: ivsc: Add CSI submodule
Quoted 6 lines
-----Original Message-----
From: Hillf Danton <[email protected]>
Sent: Friday, February 17, 2023 2:28 PM
Quoted 6 lines
On Fri, 17 Feb 2023 05:52:38 +0000 Wentong Wu <[email protected]>
Quoted 3 lines
Thanks, but what do you mean "emulated"?=20
Construct a scenario that ensures wakeup comes late because of no wait for completion. And we can see what will happen with the current test cases.
Thanks, I will remove reinit_completion
Quoted 37 lines
Review
Quoted 11 lines
static int
mei_csi_send(u8 *buf, size_t len)(quoted diff) +9
-0+{ + struct csi_cmd *cmd = (struct csi_cmd *)buf; + int ret; + + reinit_completion(&csi->cmd_completion); + + ret = mei_cldev_send(csi->cldev, buf, len); + if (ret < 0) + goto out;return -ETIMEDOUT;Quoted 23 lines
(quoted diff) +23
-0+ + ret = wait_for_completion_killable_timeout(&csi->cmd_completion, + CSI_CMD_TIMEOUT); + if (ret < 0) { + goto out; + } else if (!ret) { + ret = -ETIMEDOUT; + goto out; + } + + /* command response status */ + ret = csi->cmd_response.status; + if (ret) { + ret = -EINVAL; + goto out; + } + + if (csi->cmd_response.cmd_id != cmd->cmd_id) + ret = -EINVAL; + +out: + return ret; +}
Hello Wentong,
Quoted 5 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
Quoted 8 lines
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up system
Do you have plan to support these features in the ivsc driver in the future ?
Quoted 17 lines
Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware modules are:
- ACE (Algorithm Context Engine): This module is for algorithm computing
when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.
Quoted 4 lines
IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.
How does the privacy mode work, and how can the user trust that the closed-source IVSC and IME firmwares will honour the privacy settings ?
Quoted 4 lines
In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
Quoted 9 lines
Implementation:
There are two different drivers to handle ACE and CSI hardware modules inside IVSC.
-mei_csi: MEI client driver to send commands and receive notifications
from CSI module.
-mei_ace: MEI client driver to send commands and get status from ACE
module.
Interface is exposed viaivsc.hto acquire and release camera sensor and CSI-2 link.
Do I understand correctly, from your diagram below, that the communication between the IME and IVSC goes through SPI ?
Quoted 61 lines
Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
-----------------------------------------------------------------------------| Host Processor | | | | ----------------- ----------------- --------------- | | | | | | | | I2C | | | IPU | | ISH | |camera driver|--| | | | | | | | | | | | ----------------- ----------------- --------------- | | | | | | | | | | | --------------- | | | | | | | | | | | | | IVSC driver | | | | | | | | | | | | | --------------- | | | | | | | | ----------------|-----------------------|----------------------|---------|--- | CSI | I2C |SPI | | | | | ----------------|-----------------------|---------------- | | | IVSC | | | | | | | | | | ----------------- ----------------- | | | | | | | | | | | | | CSI | | ACE | |------| | | | | | | | | | ----------------- ----------------- | | | | | I2C | | ----------------|-----------------------|---------------- | | CSI | | | | | -------------------------------- | | | I2C | | camera sensor |-----------------------------| | | --------------------------------Wentong Wu (3):
media: pci: intel: ivsc: Add CSI submodule
media: pci: intel: ivsc: Add ACE submodule
media: pci: intel: ivsc: Add acquire/release API for ivscdrivers/media/pci/Kconfig | 1 + drivers/media/pci/intel/Makefile | 2 + drivers/media/pci/intel/ivsc/Kconfig | 12 + drivers/media/pci/intel/ivsc/Makefile | 7 + drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++ drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++ drivers/media/pci/intel/ivsc/mei_ace.h | 36 ++ drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++ drivers/media/pci/intel/ivsc/mei_csi.h | 60 ++++ include/linux/ivsc.h | 74 ++++ 10 files changed, 1090 insertions(+) create mode 100644 drivers/media/pci/intel/ivsc/Kconfig create mode 100644 drivers/media/pci/intel/ivsc/Makefile create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h create mode 100644 include/linux/ivsc.h
Signature · 3 lines
Regards, Laurent Pinchart
Hi, Wentong,
Quoted 37 lines · Laurent Pinchart
Hello Wentong,
Quoted 5 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
Quoted 8 lines
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up systemDo you have plan to support these features in the ivsc driver in the future ?
Quoted 17 lines
Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware modules are:
- ACE (Algorithm Context Engine): This module is for algorithm computing
when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.
Quoted 8 lines
Quoted 4 lines
IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.
How does the privacy mode work, and how can the user trust that the closed-source IVSC and IME firmwares will honour the privacy settings ?
Continue with question from Laurent,
How IVSC handle the privacy request from user? Is there some notification mechanism to user-space? I'd have concern if IVSC driver need private callback to request back-end(e.g. ISP driver) to handle stream cutting.
Quoted 79 lines
Quoted 4 lines
In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
Quoted 9 lines
Implementation:
There are two different drivers to handle ACE and CSI hardware modules inside IVSC.
-mei_csi: MEI client driver to send commands and receive notifications
from CSI module.
-mei_ace: MEI client driver to send commands and get status from ACE
module.
Interface is exposed viaivsc.hto acquire and release camera sensor and CSI-2 link.Do I understand correctly, from your diagram below, that the communication between the IME and IVSC goes through SPI ?
Quoted 61 lines
Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
-----------------------------------------------------------------------------| Host Processor | | | | ----------------- ----------------- --------------- | | | | | | | | I2C | | | IPU | | ISH | |camera driver|--| | | | | | | | | | | | ----------------- ----------------- --------------- | | | | | | | | | | | --------------- | | | | | | | | | | | | | IVSC driver | | | | | | | | | | | | | --------------- | | | | | | | | ----------------|-----------------------|----------------------|---------|--- | CSI | I2C |SPI | | | | | ----------------|-----------------------|---------------- | | | IVSC | | | | | | | | | | ----------------- ----------------- | | | | | | | | | | | | | CSI | | ACE | |------| | | | | | | | | | ----------------- ----------------- | | | | | I2C | | ----------------|-----------------------|---------------- | | CSI | | | | | -------------------------------- | | | I2C | | camera sensor |-----------------------------| | | --------------------------------Wentong Wu (3):
media: pci: intel: ivsc: Add CSI submodule
media: pci: intel: ivsc: Add ACE submodule
media: pci: intel: ivsc: Add acquire/release API for ivscdrivers/media/pci/Kconfig | 1 + drivers/media/pci/intel/Makefile | 2 + drivers/media/pci/intel/ivsc/Kconfig | 12 + drivers/media/pci/intel/ivsc/Makefile | 7 + drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++ drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++ drivers/media/pci/intel/ivsc/mei_ace.h | 36 ++ drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++ drivers/media/pci/intel/ivsc/mei_csi.h | 60 ++++ include/linux/ivsc.h | 74 ++++ 10 files changed, 1090 insertions(+) create mode 100644 drivers/media/pci/intel/ivsc/Kconfig create mode 100644 drivers/media/pci/intel/ivsc/Makefile create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h create mode 100644 include/linux/ivsc.h
Signature · 2 lines
Best regards, Bingbu Cao
Sakari Ailus
maintainerHi Bingbu, Wentong,
Quoted 50 lines · Bingbu Cao
Hi, Wentong,
Quoted 37 lines · Laurent Pinchart
Hello Wentong,
Quoted 5 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
Quoted 8 lines
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up systemDo you have plan to support these features in the ivsc driver in the future ?
Quoted 17 lines
Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware modules are:
- ACE (Algorithm Context Engine): This module is for algorithm computing
when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.Quoted 8 lines
Quoted 4 lines
IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.
How does the privacy mode work, and how can the user trust that the closed-source IVSC and IME firmwares will honour the privacy settings ?
Quoted 5 lines
Continue with question from Laurent,
How IVSC handle the privacy request from user? Is there some notification mechanism to user-space? I'd have concern if IVSC driver need private callback to request back-end(e.g. ISP driver) to handle stream cutting.
How does the privacy mode work, does it just pass zeroes (or other dummy data) towards the host or nothing?
A V4L2 control can be used for the purpose of passing the information to the user space at least.
Signature · 3 lines
Regards, Sakari Ailus
Hi, Sakari,
Quoted 51 lines · Sakari Ailus
Hi Bingbu, Wentong,
Quoted 48 lines · Bingbu Cao
Hi, Wentong,
Quoted 44 lines · Laurent Pinchart
Hello Wentong,
Quoted 5 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
Quoted 8 lines
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up systemDo you have plan to support these features in the ivsc driver in the future ?
Quoted 17 lines
Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware modules are:
- ACE (Algorithm Context Engine): This module is for algorithm computing
when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.Quoted 4 lines
IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.
How does the privacy mode work, and how can the user trust that the closed-source IVSC and IME firmwares will honour the privacy settings ?
As I know, without IVSC, once user enable the privacy mode, the Intel Converged Security Engine will configure the IPU camera mask (security register), which will mask the specific CSI2 port and produce dummy imaging data. For the case with IVSC, there is no final solution on Linux so far I think.
Wentong, is IVSC trying to cut off the stream and then notify user and IPU?
Quoted 14 lines
Quoted 7 lines
Quoted 1 line
Continue with question from Laurent,
How IVSC handle the privacy request from user? Is there some notification mechanism to user-space? I'd have concern if IVSC driver need private callback to request back-end(e.g. ISP driver) to handle stream cutting.
How does the privacy mode work, does it just pass zeroes (or other dummy data) towards the host or nothing?
A V4L2 control can be used for the purpose of passing the information to the user space at least.
Signature · 2 lines
Best regards, Bingbu Cao
Quoted 64 lines
-----Original Message-----
From: Bingbu Cao <[email protected]>
Sent: Friday, February 17, 2023 9:44 AMHi, Sakari,
Quoted 39 lines · Sakari Ailus
Hi Bingbu, Wentong,
Quoted 36 lines · Bingbu Cao
Hi, Wentong,
Quoted 32 lines · Laurent Pinchart
Hello Wentong,
Quoted 5 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
Quoted 9 lines
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
systemDo you have plan to support these features in the ivsc driver in the future ?
Quoted 11 lines
Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate.
These hardware modules are:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownershipof camera sensor.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
- CSI (Camera Serial Interface): This module is used to route
camera sensor data either to IVSC or to host for IPU driver and application.IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.
How does the privacy mode work, and how can the user trust that the closed-source IVSC and IME firmwares will honour the privacy settings ?
Quoted 6 lines
As I know, without IVSC, once user enable the privacy mode, the Intel Converged Security Engine will configure the IPU camera mask (security register), which will mask the specific CSI2 port and produce dummy imaging data. For the case with IVSC, there is no final solution on Linux so far I think.
Wentong, is IVSC trying to cut off the stream and then notify user and IPU?
yes
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Continue with question from Laurent,
How IVSC handle the privacy request from user? Is there some notification mechanism to user-space?
IVSC has already defined privacy callback for host IPU/camera driver.
Quoted 7 lines
Quoted 2 lines
I'd have concern if IVSC driver
Quoted 1 line
need private callback to request back-end(e.g. ISP driver) to handle stream
cutting.
Quoted 4 lines
How does the privacy mode work, does it just pass zeroes (or other dummy
data) towards the host or nothing?
No data on CSI transmitter side
Quoted 3 lines
Quoted 3 lines
A V4L2 control can be used for the purpose of passing the information to the user space at least.
I will take some time to review V4L2 sub-device and control mechanism, and then update the driver.
BR,
Wentong
Quoted 5 lines
Quoted 1 line
--
Best regards,
Bingbu Cao
Sakari Ailus
maintainerHi Wentong,
Quoted 64 lines · Wentong
Quoted 64 lines
-----Original Message-----
From: Bingbu Cao <[email protected]>
Sent: Friday, February 17, 2023 9:44 AMHi, Sakari,
Quoted 39 lines · Sakari Ailus
Hi Bingbu, Wentong,
Quoted 36 lines · Bingbu Cao
Hi, Wentong,
Quoted 32 lines · Laurent Pinchart
Hello Wentong,
Quoted 5 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
Quoted 9 lines
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
systemDo you have plan to support these features in the ivsc driver in the future ?
Quoted 11 lines
Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate.
These hardware modules are:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownershipof camera sensor.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
- CSI (Camera Serial Interface): This module is used to route
camera sensor data either to IVSC or to host for IPU driver and application.IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.
How does the privacy mode work, and how can the user trust that the closed-source IVSC and IME firmwares will honour the privacy settings ?
Quoted 8 lines
Quoted 6 lines
As I know, without IVSC, once user enable the privacy mode, the Intel Converged Security Engine will configure the IPU camera mask (security register), which will mask the specific CSI2 port and produce dummy imaging data. For the case with IVSC, there is no final solution on Linux so far I think.
Wentong, is IVSC trying to cut off the stream and then notify user and IPU?
yes
Does the CSI-2 transmitter on IVCS go to some LP mode during this time, or does the receiver need to initialise the bus again when the stream resuems?
Quoted 12 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Continue with question from Laurent,
How IVSC handle the privacy request from user? Is there some notification mechanism to user-space?
IVSC has already defined privacy callback for host IPU/camera driver.
Quoted 1 line
Quoted 1 line
I'd have concern if IVSC driver
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
need private callback to request back-end(e.g. ISP driver) to handle stream
cutting.
Quoted 4 lines
How does the privacy mode work, does it just pass zeroes (or other dummy
data) towards the host or nothing?No data on CSI transmitter side
Can it stop in the middle of the frame? Or is it guaranteed to produce full frames (assuming the sensor does)?
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
A V4L2 control can be used for the purpose of passing the information to the user space at least.
I will take some time to review V4L2 sub-device and control mechanism, and then update the driver.
Signature · 3 lines
Kind regards, Sakari Ailus
Hi Sakari,
Quoted 83 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Friday, February 17, 2023 6:50 PMHi Wentong,
Quoted 39 lines · Wentong
Quoted 39 lines
-----Original Message-----
From: Bingbu Cao <[email protected]>
Sent: Friday, February 17, 2023 9:44 AMHi, Sakari,
Quoted 30 lines · Sakari Ailus
Hi Bingbu, Wentong,
Quoted 27 lines · Bingbu Cao
Hi, Wentong,
Quoted 23 lines · Laurent Pinchart
Hello Wentong,
Quoted 5 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
Quoted 11 lines
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a
CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor
Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking
up systemDo you have plan to support these features in the ivsc driver in the future ?
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces
provided by this IVSC driver.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to
enumerate.
Quoted 8 lines
Quoted 8 lines
These hardware modules are:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownershipof camera sensor.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
- CSI (Camera Serial Interface): This module is used to route
camera sensor data either to IVSC or to host for IPU driver andapplication.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be
notified.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
How does the privacy mode work, and how can the user trust that the closed-source IVSC and IME firmwares will honour the privacy
settings ?
Quoted 5 lines
Quoted 5 lines
As I know, without IVSC, once user enable the privacy mode, the Intel Converged Security Engine will configure the IPU camera mask (security register), which will mask the specific CSI2 port and produce dummy imaging data. For the case with IVSC, there is no final
Quoted 7 lines
solution on Linux so far I think.
Quoted 4 lines
Quoted 2 lines
Wentong, is IVSC trying to cut off the stream and then notify user and IPU?
yes
Does the CSI-2 transmitter on IVCS go to some LP mode during this time,
Yes, The low power mode is following the D-Phy spec.
Quoted 2 lines
or does
the receiver need to initialise the bus again when the stream resuems?
No, it's the same link.
Quoted 17 lines
Quoted 17 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 4 lines
Continue with question from Laurent,
How IVSC handle the privacy request from user? Is there some notification mechanism to user-space?
IVSC has already defined privacy callback for host IPU/camera driver.
Quoted 5 lines
Quoted 3 lines
I'd have concern if IVSC driver
Quoted 2 lines
need private callback to request back-end(e.g. ISP driver) to handle stream
cutting.
Quoted 1 line
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
How does the privacy mode work, does it just pass zeroes (or other dummy
data) towards the host or nothing?No data on CSI transmitter side
Can it stop in the middle of the frame?
No,
Or is it guaranteed to produce full frames
Quoted 1 line
(assuming the sensor does)?
Yes, full frame will be guaranteed as camera sensor does.
Quoted 13 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
A V4L2 control can be used for the purpose of passing the information to the user space at least.
I will take some time to review V4L2 sub-device and control mechanism, and then update the driver.
--
Kind regards,Sakari Ailus
Quoted 23 lines
-----Original Message-----
From: Laurent Pinchart <[email protected]>
Sent: Wednesday, February 15, 2023 5:43 PMHello Wentong,
Quoted 5 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
Quoted 9 lines
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
system
Quoted 1 line
Do you have plan to support these features in the ivsc driver in the future ?
Not sure, but the first step is to upstream this driver.
Quoted 20 lines
Quoted 8 lines
Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.
Quoted 4 lines
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware
modules are:
Quoted 3 lines
- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership ofcamera sensor.
Quoted 3 lines
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.
Quoted 7 lines
Quoted 4 lines
IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.
How does the privacy mode work, and how can the user trust that the closed- source IVSC and IME firmwares will honour the privacy settings ?
No camera data will be allowed to go through IVSC, and then there will be no data on IVSC CSI transmitter side.
Quoted 86 lines
Quoted 5 lines
In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
Quoted 10 lines
Implementation:
There are two different drivers to handle ACE and CSI hardware modules inside IVSC.
-mei_csi: MEI client driver to send commands and receive
notifications from CSI module.
-mei_ace: MEI client driver to send commands and get status from ACE
module.
Interface is exposed viaivsc.hto acquire and release camera sensor and
CSI-2 link.Do I understand correctly, from your diagram below, that the communication between the IME and IVSC goes through SPI ?
Quoted 54 lines
Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
----------------------------------------------------------------------
-------| Host Processor | | | | ----------------- ----------------- --------------- | | | | | | | | I2C | | | IPU | | ISH | |camera driver|--| | | | | | | | | | | | ----------------- ----------------- --------------- | | | | | | | | | | | --------------- | | | | | | | | | | | | | IVSC driver | | | | | | | | | | | | | --------------- | | | | | | | | ----------------|-----------------------|----------------------|---------|--- | CSI | I2C |SPI | | | | | ----------------|-----------------------|---------------- | | | IVSC | | | | | | | | | | ----------------- ----------------- | | | | | | | | | | | | | CSI | | ACE | |------| | | | | | | | | | ----------------- ----------------- | | | | | I2C | | ----------------|-----------------------|---------------- | | CSI | | | | | -------------------------------- | | | I2C | | camera sensor |-----------------------------| | | --------------------------------Wentong Wu (3):
media: pci: intel: ivsc: Add CSI submodule
media: pci: intel: ivsc: Add ACE submodule
media: pci: intel: ivsc: Add acquire/release API for ivscdrivers/media/pci/Kconfig | 1 + drivers/media/pci/intel/Makefile | 2 + drivers/media/pci/intel/ivsc/Kconfig | 12 + drivers/media/pci/intel/ivsc/Makefile | 7 + drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++ drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++ drivers/media/pci/intel/ivsc/mei_ace.h | 36 ++ drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++ drivers/media/pci/intel/ivsc/mei_csi.h | 60 ++++ include/linux/ivsc.h | 74 ++++ 10 files changed, 1090 insertions(+)Quoted 8 lines
create mode 100644 drivers/media/pci/intel/ivsc/Kconfig create mode 100644 drivers/media/pci/intel/ivsc/Makefile create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h create mode 100644 include/linux/ivsc.h--
Regards,Laurent Pinchart
Hello Wentong,
Quoted 17 lines · Wentong
Quoted 16 lines · Laurent Pinchart
Quoted 14 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
system
Quoted 3 lines
Quoted 1 line
Do you have plan to support these features in the ivsc driver in the future ?
Not sure, but the first step is to upstream this driver.
Sure, no problem.
Quoted 24 lines
Quoted 24 lines
Quoted 23 lines
Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware modules are:
- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.
Quoted 5 lines
Quoted 2 lines
How does the privacy mode work, and how can the user trust that the closed- source IVSC and IME firmwares will honour the privacy settings ?
No camera data will be allowed to go through IVSC, and then there will be no data on IVSC CSI transmitter side.
But how can I be sure that the IVSC will not use the camera behind my back, if it's all controlled through a closed-source firmware ?
Quoted 76 lines
Quoted 76 lines
Quoted 11 lines
In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
Implementation:
There are two different drivers to handle ACE and CSI hardware modules inside IVSC.
-mei_csi: MEI client driver to send commands and receive notifications from CSI module.
-mei_ace: MEI client driver to send commands and get status from ACE module.
Interface is exposed viaivsc.hto acquire and release camera sensor and CSI-2 link.Do I understand correctly, from your diagram below, that the communication between the IME and IVSC goes through SPI ?
Quoted 61 lines
Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
-----------------------------------------------------------------------------| Host Processor | | | | ----------------- ----------------- --------------- | | | | | | | | I2C | | | IPU | | ISH | |camera driver|--| | | | | | | | | | | | ----------------- ----------------- --------------- | | | | | | | | | | | --------------- | | | | | | | | | | | | | IVSC driver | | | | | | | | | | | | | --------------- | | | | | | | | ----------------|-----------------------|----------------------|---------|--- | CSI | I2C |SPI | | | | | ----------------|-----------------------|---------------- | | | IVSC | | | | | | | | | | ----------------- ----------------- | | | | | | | | | | | | | CSI | | ACE | |------| | | | | | | | | | ----------------- ----------------- | | | | | I2C | | ----------------|-----------------------|---------------- | | CSI | | | | | -------------------------------- | | | I2C | | camera sensor |-----------------------------| | | --------------------------------Wentong Wu (3):
media: pci: intel: ivsc: Add CSI submodule
media: pci: intel: ivsc: Add ACE submodule
media: pci: intel: ivsc: Add acquire/release API for ivscdrivers/media/pci/Kconfig | 1 + drivers/media/pci/intel/Makefile | 2 + drivers/media/pci/intel/ivsc/Kconfig | 12 + drivers/media/pci/intel/ivsc/Makefile | 7 + drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++ drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++ drivers/media/pci/intel/ivsc/mei_ace.h |> 36 ++ drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++ drivers/media/pci/intel/ivsc/mei_csi.h | 60 ++++ include/linux/ivsc.h | 74 ++++ 10 files changed, 1090 insertions(+) create mode 100644 drivers/media/pci/intel/ivsc/Kconfig create mode 100644 drivers/media/pci/intel/ivsc/Makefile create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h create mode 100644 include/linux/ivsc.h
Signature · 3 lines
Regards, Laurent Pinchart
Quoted 57 lines
-----Original Message-----
From: Laurent Pinchart <[email protected]>
Sent: Friday, February 17, 2023 7:12 PMHello Wentong,
Quoted 17 lines · Wentong
Quoted 16 lines · Laurent Pinchart
Quoted 14 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
systemQuoted 3 lines
Quoted 1 line
Do you have plan to support these features in the ivsc driver in the future ?
Not sure, but the first step is to upstream this driver.
Sure, no problem.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by
this IVSC driver.
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware modules are:
- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route
camera sensor data either to IVSC or to host for IPU driver and application.IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
How does the privacy mode work, and how can the user trust that the closed- source IVSC and IME firmwares will honour the privacy settings ?
No camera data will be allowed to go through IVSC, and then there will be no data on IVSC CSI transmitter side.
But how can I be sure that the IVSC will not use the camera behind my back, if it's all controlled through a closed-source firmware ?
Actually I don't know how to answer your question, but this is guaranteed though we have no plan to open-source the firmware.
Quoted 86 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
Implementation:
There are two different drivers to handle ACE and CSI hardware modules inside IVSC.
-mei_csi: MEI client driver to send commands and receive notificationsfrom CSI module.
Quoted 1 line
Quoted 1 line
Quoted 1 line
-
mei_ace: MEI client driver to send commands and get status from ACEmodule.
Quoted 61 lines
Quoted 61 lines
Quoted 3 lines
Interface is exposed via
ivsc.hto acquire and release camera sensor and
CSI-2 link.Do I understand correctly, from your diagram below, that the communication between the IME and IVSC goes through SPI ?
Quoted 54 lines
Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
------------------------------------------------------------------
-----------| Host Processor | | | | ----------------- ----------------- --------------- | | | | | | | | I2C | | | IPU | | ISH | |camera driver|--| | | | | | | | | | | | ----------------- ----------------- --------------- | | | | | | | | | | | --------------- | | | | | | | | | | | | | IVSC driver | | | | | | | | | | | | | --------------- | | | | | | | | ----------------|-----------------------|----------------------|---------|--- | CSI | I2C |SPI | | | | | ----------------|-----------------------|---------------- | | | IVSC | | | | | | | | | | ----------------- ----------------- | | | | | | | | | | | | | CSI | | ACE | |------| | | | | | | | | | ----------------- ----------------- | | | | | I2C | | ----------------|-----------------------|---------------- | | CSI | | | | | -------------------------------- | | | I2C | | camera sensor |-----------------------------| | | --------------------------------Wentong Wu (3):
media: pci: intel: ivsc: Add CSI submodule
media: pci: intel: ivsc: Add ACE submodule
media: pci: intel: ivsc: Add acquire/release API for ivscdrivers/media/pci/Kconfig | 1 + drivers/media/pci/intel/Makefile | 2 + drivers/media/pci/intel/ivsc/Kconfig | 12 + drivers/media/pci/intel/ivsc/Makefile | 7 + drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++ drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++ drivers/media/pci/intel/ivsc/mei_ace.h |> 36 ++ drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++ drivers/media/pci/intel/ivsc/mei_csi.h | 60 ++++ include/linux/ivsc.h | 74 ++++ 10 files changed, 1090 insertions(+) create mode 100644Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/media/pci/intel/ivsc/Kconfigcreate mode 100644 drivers/media/pci/intel/ivsc/Makefile create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h create mode 100644 include/linux/ivsc.h--
Regards,Laurent Pinchart
Sakari Ailus
maintainerHi Wentong,
Quoted 31 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up systemSince the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware modules are:
- ACE (Algorithm Context Engine): This module is for algorithm computing
when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.
Quoted 8 lines
IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.
In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
I thought this for a while and did some research, and I can suggest the following:
- The IVSC sub-device implements a control for privacy (V4L2_CID_PRIVACY is a good fit).
- Camera sensor access needs to be requested from IVSC before accessing the sensor via I²C. The IVSC ownership control needs to be in the right setting for this to work, and device links can be used for that purpose (see device_link_add()). With DL_FLAG_PM_RUNTIME and DL_FLAG_RPM_ACTIVE, the supplier devices will be PM runtime resumed before the consumer (camera sensor). As these devices are purely virtual on host side and has no power state as such, you can use runtime PM callbacks to transfer the ownership.
- The CSI-2 configuration should take place when streaming starts on the sensor (as well as IVSC).
- Device links need to be set up via IPU bridge which now is called CIO2 bridge (cio2-bridge.c).
Any questions, comments?
Signature · 3 lines
Kind regards, Sakari Ailus
Hi,
Quoted 49 lines · Sakari Ailus
Hi Wentong,
Quoted 31 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up systemSince the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware modules are:
- ACE (Algorithm Context Engine): This module is for algorithm computing
when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.Quoted 8 lines
IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.
In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
I thought this for a while and did some research, and I can suggest the following:
- The IVSC sub-device implements a control for privacy (
V4L2_CID_PRIVACYis a good fit).
Quoted 8 lines
- Camera sensor access needs to be requested from IVSC before accessing the sensor via I²C. The IVSC ownership control needs to be in the right setting for this to work, and device links can be used for that purpose (see
device_link_add()). WithDL_FLAG_PM_RUNTIMEandDL_FLAG_RPM_ACTIVE, the supplier devices will be PM runtime resumed before the consumer (camera sensor). As these devices are purely virtual on host side and has no power state as such, you can use runtime PM callbacks to transfer the ownership.
Interesting proposal to use device-links + runtime-pm for this instead of modelling this as an i2c-mux. FWIW I'm fine with going this route instead of using an i2c-mux approach.
I have been thinking about the i2c-mux approach a bit and the problem is that we are not really muxing but want to turn on/off control and AFAIK the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain, so control will never be released when the i2c transfers are done.
And if were to somehow modify things (or maybe there already is some release callback) then the downside becomes that the i2c-mux core code operates at the i2c transfer level. So each i2c read/write would then enable + disavle control.
Modelling this using something like runtime pm as such is a much better fit because then we request control once on probe / stream-on and release it once we are fully done, rather then requesting + releasing control once per i2c-transfer.
Regards,
Hans
Quoted 9 lines
- The CSI-2 configuration should take place when streaming starts on the sensor (as well as IVSC).
- Device links need to be set up via IPU bridge which now is called CIO2 bridge (
cio2-bridge.c).Any questions, comments?
Quoted 81 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Wednesday, March 1, 2023 6:42 PMHi,
Quoted 29 lines · Sakari Ailus
Hi Wentong,
Quoted 26 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
systemSince the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware
modules are:
Quoted 3 lines
Quoted 3 lines
- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownershipof camera sensor.
Quoted 16 lines
Quoted 11 lines
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.
In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
I thought this for a while and did some research, and I can suggest the
following:Quoted 7 lines
- The IVSC sub-device implements a control for privacy (
V4L2_CID_PRIVACYis a good fit).- Camera sensor access needs to be requested from IVSC before accessing the sensor via I²C. The IVSC ownership control needs to be in the right setting for this to work, and device links can be used for that purpose (see
device_link_add()). WithDL_FLAG_PM_RUNTIMEand
DL_FLAG_RPM_ACTIVE,Quoted 4 lines
the supplier devices will be PM runtime resumed before the consumer (camera sensor). As these devices are purely virtual on host side and has no power state as such, you can use runtime PM callbacks to transfer the ownership.
Interesting proposal to use device-links + runtime-pm for this instead of modelling this as an i2c-mux. FWIW I'm fine with going this route instead of using an i2c-mux approach.
I have been thinking about the i2c-mux approach a bit and the problem is that we are not really muxing but want to turn on/off control and AFAIK the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain, so control will never be released when the i2c transfers are done.
Quoted 8 lines
And if were to somehow modify things (or maybe there already is some release callback) then the downside becomes that the i2c-mux core code operates at the i2c transfer level. So each i2c read/write would then enable + disavle control.
Modelling this using something like runtime pm as such is a much better fit because then we request control once on probe / stream-on and release it once we are fully done, rather then requesting + releasing control once per i2c- transfer.
Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe, probably we have to switch to i2c-mux modeling way.
Thanks
Wentong
Quoted 16 lines
Regards,
Hans
Quoted 9 lines
- The CSI-2 configuration should take place when streaming starts on the sensor (as well as IVSC).
- Device links need to be set up via IPU bridge which now is called CIO2 bridge (
cio2-bridge.c).Any questions, comments?
Sakari Ailus
maintainerHi Wentong,
Quoted 85 lines · Wentong
Quoted 85 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Wednesday, March 1, 2023 6:42 PMHi,
Quoted 29 lines · Sakari Ailus
Hi Wentong,
Quoted 26 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
systemSince the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware
modules are:
Quoted 3 lines
Quoted 3 lines
- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownershipof camera sensor.
Quoted 16 lines
Quoted 11 lines
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.
In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
I thought this for a while and did some research, and I can suggest the
following:Quoted 7 lines
- The IVSC sub-device implements a control for privacy (
V4L2_CID_PRIVACYis a good fit).- Camera sensor access needs to be requested from IVSC before accessing the sensor via I²C. The IVSC ownership control needs to be in the right setting for this to work, and device links can be used for that purpose (see
device_link_add()). WithDL_FLAG_PM_RUNTIMEand
DL_FLAG_RPM_ACTIVE,Quoted 4 lines
the supplier devices will be PM runtime resumed before the consumer (camera sensor). As these devices are purely virtual on host side and has no power state as such, you can use runtime PM callbacks to transfer the ownership.
Interesting proposal to use device-links + runtime-pm for this instead of modelling this as an i2c-mux. FWIW I'm fine with going this route instead of using an i2c-mux approach.
I have been thinking about the i2c-mux approach a bit and the problem is that we are not really muxing but want to turn on/off control and AFAIK the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain, so control will never be released when the i2c transfers are done.
And if were to somehow modify things (or maybe there already is some release callback) then the downside becomes that the i2c-mux core code operates at the i2c transfer level. So each i2c read/write would then enable + disavle control.
Quoted 7 lines
Quoted 4 lines
Modelling this using something like runtime pm as such is a much better fit because then we request control once on probe / stream-on and release it once we are fully done, rather then requesting + releasing control once per i2c- transfer.
Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe, probably we have to switch to i2c-mux modeling way.
What do you mean? The supplier devices are resumed before the driver's probe is called.
Signature · 3 lines
Regards, Sakari Ailus
Quoted 108 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Tuesday, March 7, 2023 4:30 PMHi Wentong,
Quoted 27 lines · Wentong
Quoted 27 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Wednesday, March 1, 2023 6:42 PMHi,
Quoted 18 lines · Sakari Ailus
Hi Wentong,
Quoted 15 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
systemQuoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided
by this IVSC driver.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware
Quoted 8 lines
Quoted 8 lines
modules are:
Quoted 4 lines
Quoted 4 lines
- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownershipof camera sensor.
Quoted 2 lines
Quoted 2 lines
- CSI (Camera Serial Interface): This module is used to route
camera sensor data either to IVSC or to host for IPU driver andapplication.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be
notified.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
I thought this for a while and did some research, and I can suggest the
following:- The IVSC sub-device implements a control for privacy (
V4L2_CID_PRIVACYis a good fit).- Camera sensor access needs to be requested from IVSC before accessing
the
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
sensor via I²C. The IVSC ownership control needs to be in the right setting for this to work, and device links can be used for that purpose (see
device_link_add()). WithDL_FLAG_PM_RUNTIMEand
DL_FLAG_RPM_ACTIVE,Quoted 4 lines
the supplier devices will be PM runtime resumed before the consumer (camera sensor). As these devices are purely virtual on host side and has no power state as such, you can use runtime PM callbacks to transfer the ownership.
Quoted 8 lines
Quoted 8 lines
Interesting proposal to use device-links + runtime-pm for this instead of modelling this as an i2c-mux. FWIW I'm fine with going this route instead of using an i2c-mux approach.
I have been thinking about the i2c-mux approach a bit and the problem is that we are not really muxing but want to turn on/off control and AFAIK the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain, so control will never be released when the i2c
transfers are done.
Quoted 5 lines
Quoted 5 lines
And if were to somehow modify things (or maybe there already is some release
callback) then the downside becomes that the i2c-mux core code operates at the i2c transfer level. So each i2c read/write would then enable +disavle control.
Quoted 6 lines
Quoted 5 lines
Modelling this using something like runtime pm as such is a much better fit because then we request control once on probe / stream-on and release it once we are fully done, rather then requesting + releasing control once per i2c- transfer.
Quoted 5 lines
Quoted 2 lines
Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe, probably we have to switch to i2c-mux modeling way.
What do you mean? The supplier devices are resumed before the driver's probe is called.
But we setup the link with device_link_add during IVSC driver's probe, we can't guarantee driver probe's sequence.
Quoted 5 lines
--
Regards,Sakari Ailus
Hi,
Quoted 108 lines · Wentong
Quoted 108 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Tuesday, March 7, 2023 4:30 PMHi Wentong,
Quoted 27 lines · Wentong
Quoted 27 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Wednesday, March 1, 2023 6:42 PMHi,
Quoted 18 lines · Sakari Ailus
Hi Wentong,
Quoted 15 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
systemQuoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided
by this IVSC driver.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware
Quoted 8 lines
Quoted 8 lines
modules are:
Quoted 4 lines
Quoted 4 lines
- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownershipof camera sensor.
Quoted 2 lines
Quoted 2 lines
- CSI (Camera Serial Interface): This module is used to route
camera sensor data either to IVSC or to host for IPU driver andapplication.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be
notified.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
I thought this for a while and did some research, and I can suggest the
following:- The IVSC sub-device implements a control for privacy (
V4L2_CID_PRIVACYis a good fit).- Camera sensor access needs to be requested from IVSC before accessing
the
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
sensor via I²C. The IVSC ownership control needs to be in the right setting for this to work, and device links can be used for that purpose (see
device_link_add()). WithDL_FLAG_PM_RUNTIMEand
DL_FLAG_RPM_ACTIVE,Quoted 4 lines
the supplier devices will be PM runtime resumed before the consumer (camera sensor). As these devices are purely virtual on host side and has no power state as such, you can use runtime PM callbacks to transfer the ownership.
Quoted 8 lines
Quoted 8 lines
Interesting proposal to use device-links + runtime-pm for this instead of modelling this as an i2c-mux. FWIW I'm fine with going this route instead of using an i2c-mux approach.
I have been thinking about the i2c-mux approach a bit and the problem is that we are not really muxing but want to turn on/off control and AFAIK the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain, so control will never be released when the i2c
transfers are done.
Quoted 5 lines
Quoted 5 lines
And if were to somehow modify things (or maybe there already is some release
callback) then the downside becomes that the i2c-mux core code operates at the i2c transfer level. So each i2c read/write would then enable +disavle control.
Quoted 6 lines
Quoted 5 lines
Modelling this using something like runtime pm as such is a much better fit because then we request control once on probe / stream-on and release it once we are fully done, rather then requesting + releasing control once per i2c- transfer.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe, probably we have to switch to i2c-mux modeling way.
What do you mean? The supplier devices are resumed before the driver's probe is called.
But we setup the link with
device_link_addduring IVSC driver's probe, we can't guarantee driver probe's sequence.
Then maybe we need to do the device_link_add somewhere else.
The mainline kernel delays probing of camera sensors on Intel platforms until the INT3472 driver has probed the INT3472 device on which the sensors have an ACPI _DEP.
This is already used to make sure that clock lookups and regulator info is in place before the sensor's probe() function runs.
So that when the driver does clk_get() it succeeds and so that regulator_get() does not end up returning a dummy regulator.
So I think the code adding the device_link-s for the IVSC should be added
to: drivers/platform/x86/intel/int3472/discrete.c and then the runtime-resume will happen before the sensor's probe() function runs.
Likewise drivers/platform/x86/intel/int3472/discrete.c should also ensure that the ivsc driver's probe() has run before it calls acpi_dev_clear_dependencies().
The acpi_dev_clear_dependencies() call in discrete.c tells the ACPI subsystem to go ahead and create the i2c-clients for the sensors and allow the sensor drivers to get loaded and probe the sensor.
Regards,
Hans
Quoted 128 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Tuesday, March 7, 2023 5:10 PMHi,
Quoted 71 lines · Wentong
Quoted 71 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Tuesday, March 7, 2023 4:30 PMHi Wentong,
Quoted 27 lines · Wentong
Quoted 27 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Wednesday, March 1, 2023 6:42 PMHi,
Quoted 18 lines · Sakari Ailus
Hi Wentong,
Quoted 15 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
systemQuoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided
by this IVSC driver.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware
Quoted 8 lines
Quoted 8 lines
modules are:
Quoted 4 lines
Quoted 4 lines
- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownershipof camera sensor.
Quoted 2 lines
Quoted 2 lines
- CSI (Camera Serial Interface): This module is used to route
camera sensor data either to IVSC or to host for IPU driver andapplication.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be
notified.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
I thought this for a while and did some research, and I can suggest the
following:- The IVSC sub-device implements a control for privacy
(
V4L2_CID_PRIVACYQuoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
is a good fit).
- Camera sensor access needs to be requested from IVSC before accessing
Quoted 8 lines
Quoted 8 lines
the
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
sensor via I²C. The IVSC ownership control needs to be in the right setting for this to work, and device links can be used for that purpose (see
device_link_add()). WithDL_FLAG_PM_RUNTIMEand
DL_FLAG_RPM_ACTIVE,Quoted 3 lines
the supplier devices will be PM runtime resumed before the consumer (camera sensor). As these devices are purely virtual on host side and has no power state as such, you can use runtime PM callbacks to transfer
the
Quoted 20 lines
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
Quoted 1 line
ownership.
Interesting proposal to use device-links + runtime-pm for this instead of modelling this as an i2c-mux. FWIW I'm fine with going this route instead of using an i2c-mux approach.
I have been thinking about the i2c-mux approach a bit and the problem is that we are not really muxing but want to turn on/off control and AFAIK the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain, so control will never be released when the i2c
transfers are done.
Quoted 6 lines
Quoted 6 lines
And if were to somehow modify things (or maybe there already is some release
callback) then the downside becomes that the i2c-mux core code operates at the i2c transfer level. So each i2c read/write would then enable +disavle control.
Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Modelling this using something like runtime pm as such is a much better fit because then we request control once on probe / stream-on and release it once we are fully done, rather then requesting + releasing control once per i2c- transfer.
Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe, probably we have to switch to i2c-mux modeling
way.
Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
What do you mean? The supplier devices are resumed before the driver's probe is called.
But we setup the link with
device_link_addduring IVSC driver's probe, we can't guarantee driver probe's sequence.Then maybe we need to do the
device_link_addsomewhere else.
sensor's parent is the LJCA I2C device whose driver is being upstream
https://www.spinics.net/lists/kernel/msg4702552.htmland and sensor's power is controlled by IVSC instead of INT3472 if IVSC enabled.
struct device_link *device_link_add(struct device *consumer,
struct device *supplier, u32 flags)
So probably we have to add above device_link_add in LJCA I2C's driver, and we can find the consumer(camera sensor) with ACPI API, but the supplier, mei_ace, is mei client device under mei framework and it's dynamically allocated device instead of ACPI device, probably I can find its parent with some ACPI lookup from this LJCA I2C device, but unfortunately mei framework doesn't export the API to find mei client device with its parent bus device(struct mei_device).
I'm not sure if modeling this mei_ace as LJCA I2C's runtime power control is acceptable, if yes, probably this mei_ace driver have to go with LJCA I2C device driver.
BR,
Wentong
Quoted 25 lines
The mainline kernel delays probing of camera sensors on Intel platforms until the INT3472 driver has probed the INT3472 device on which the sensors have an ACPI
_DEP.This is already used to make sure that clock lookups and regulator info is in place before the sensor's
probe()function runs.So that when the driver does
clk_get()it succeeds and so thatregulator_get()does not end up returning a dummy regulator.So I think the code adding the device_link-s for the IVSC should be added
to:drivers/platform/x86/intel/int3472/discrete.cand then the runtime-resume will happen before the sensor'sprobe()function runs.Likewise
drivers/platform/x86/intel/int3472/discrete.cshould also ensure that the ivsc driver'sprobe()has run before it callsacpi_dev_clear_dependencies().The
acpi_dev_clear_dependencies()call indiscrete.ctells the ACPI subsystem to go ahead and create the i2c-clients for the sensors and allow the sensor drivers to get loaded and probe the sensor.Regards,
Hans
Hi,
Quoted 131 lines · Wentong
Quoted 131 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Tuesday, March 7, 2023 5:10 PMHi,
Quoted 71 lines · Wentong
Quoted 71 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Tuesday, March 7, 2023 4:30 PMHi Wentong,
Quoted 27 lines · Wentong
Quoted 27 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Wednesday, March 1, 2023 6:42 PMHi,
Quoted 18 lines · Sakari Ailus
Hi Wentong,
Quoted 15 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
systemQuoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided
by this IVSC driver.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware
Quoted 8 lines
Quoted 8 lines
modules are:
Quoted 4 lines
Quoted 4 lines
- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownershipof camera sensor.
Quoted 2 lines
Quoted 2 lines
- CSI (Camera Serial Interface): This module is used to route
camera sensor data either to IVSC or to host for IPU driver andapplication.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be
notified.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
I thought this for a while and did some research, and I can suggest the
following:- The IVSC sub-device implements a control for privacy
(
V4L2_CID_PRIVACYQuoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
is a good fit).
- Camera sensor access needs to be requested from IVSC before accessing
Quoted 8 lines
Quoted 8 lines
the
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
sensor via I²C. The IVSC ownership control needs to be in the right setting for this to work, and device links can be used for that purpose (see
device_link_add()). WithDL_FLAG_PM_RUNTIMEand
DL_FLAG_RPM_ACTIVE,Quoted 3 lines
the supplier devices will be PM runtime resumed before the consumer (camera sensor). As these devices are purely virtual on host side and has no power state as such, you can use runtime PM callbacks to transfer
the
Quoted 20 lines
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
Quoted 1 line
ownership.
Interesting proposal to use device-links + runtime-pm for this instead of modelling this as an i2c-mux. FWIW I'm fine with going this route instead of using an i2c-mux approach.
I have been thinking about the i2c-mux approach a bit and the problem is that we are not really muxing but want to turn on/off control and AFAIK the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain, so control will never be released when the i2c
transfers are done.
Quoted 6 lines
Quoted 6 lines
And if were to somehow modify things (or maybe there already is some release
callback) then the downside becomes that the i2c-mux core code operates at the i2c transfer level. So each i2c read/write would then enable +disavle control.
Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Modelling this using something like runtime pm as such is a much better fit because then we request control once on probe / stream-on and release it once we are fully done, rather then requesting + releasing control once per i2c- transfer.
Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe, probably we have to switch to i2c-mux modeling
way.
Quoted 4 lines
Quoted 3 lines
What do you mean? The supplier devices are resumed before the driver's probe is called.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
But we setup the link with
device_link_addduring IVSC driver's probe, we can't guarantee driver probe's sequence.Then maybe we need to do the
device_link_addsomewhere else.sensor's parent is the LJCA I2C device whose driver is being upstream
https://www.spinics.net/lists/kernel/msg4702552.htmland and sensor's power is controlled by IVSC instead of INT3472 if IVSC enabled.
I believe that the INT3472 code is still involved at least on a Dell Latitude 9420 the INT3472 code still needs to set the clock-enable and the privacy-LED GPIOs otherwise the main camera won't work.
So I'm not sure what you mean with "sensor's power is controlled by IVSC instead of INT3472" ?
Quoted 11 lines
struct
device_link*device_link_add(struct device *consumer,
struct device *supplier, u32 flags)So probably we have to add above
device_link_addin LJCA I2C's driver, and we can find the consumer(camera sensor) with ACPI API, but the supplier,mei_ace, is mei client device under mei framework and it's dynamically allocated device instead of ACPI device, probably I can find its parent with some ACPI lookup from this LJCA I2C device, but unfortunately mei framework doesn't export the API to find mei client device with its parent busdevice(struct mei_device).
Quoted 3 lines
I'm not sure if modeling this
mei_aceas LJCA I2C's runtime power control is acceptable, if yes, probably thismei_acedriver have to go with LJCA I2C device driver.
Looking at the ACPI table the sensor ACPI device has 2 _DEP-s listed the I2C controller and the INT3472 device. Since we are already doing similar setup in the INT3472 device that seems like a good place to add the device_link()-s (it can return -EPROBE_DEFER to wait for the mei_ace to show up).
But when the INT3472 code runs, the consumer device does not exist yet and AFAICT the same is true when the LCJA i2c-controller driver is getting registered. The consumer only exists when the i2c_client is instantiated and at that point the sensor drivers probe() method can run immediately and we are too late to add the device_link.
As a hobby project I have been working on atomisp2 support and I have a similar issue there. There is no INT3472 device there, but there is a _DSM method which needs to be used to figure out which ACPI GPIO resource is reset / powerdown and if the GPIOs are active-low or active high.
I have written a little helper function to call the _DSM and to then turn this into lookups and call devm_acpi_dev_add_driver_gpios().
Since on atomisp2 we cannot use the INT3472 driver to delay the sensor-driver probe and have the INT3472 driver setup the GPIO lookup, at least for the sensor drivers used with atomisp2 there is going to be a need to add a single line to probe() like this:
v4l2_get_acpi_sensor_info(&i2c_client->dev, NULL);
To me it sounds like we need to do something similar here and extend the helper function which I have written (but not yet submitted upstream) :
https://github.com/jwrdegoede/linux-sunxi/commit/e2287979db43d46fa7d354c1bde92eb6219b613d
To also setup the device-links needed for the runtime-pm solution to getting the i2c passed through to the sensor.
Ideally v4l2_get_acpi_sensor_info() should return void (easier to use in the sensor drivers) but I think it should return an int, so that it can e.g. return -EPROBE_DEFER to wait for the mei_ace.
Regards,
Hans
Quoted 25 lines
Quoted 24 lines
The mainline kernel delays probing of camera sensors on Intel platforms until the INT3472 driver has probed the INT3472 device on which the sensors have an ACPI
_DEP.This is already used to make sure that clock lookups and regulator info is in place before the sensor's
probe()function runs.So that when the driver does
clk_get()it succeeds and so thatregulator_get()does not end up returning a dummy regulator.So I think the code adding the device_link-s for the IVSC should be added
to:drivers/platform/x86/intel/int3472/discrete.cand then the runtime-resume will happen before the sensor'sprobe()function runs.Likewise
drivers/platform/x86/intel/int3472/discrete.cshould also ensure that the ivsc driver'sprobe()has run before it callsacpi_dev_clear_dependencies().The
acpi_dev_clear_dependencies()call indiscrete.ctells the ACPI subsystem to go ahead and create the i2c-clients for the sensors and allow the sensor drivers to get loaded and probe the sensor.Regards,
Hans
Hi
Quoted 150 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 5:28 PMHi,
Quoted 90 lines · Wentong
Quoted 90 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Tuesday, March 7, 2023 5:10 PMHi,
Quoted 71 lines · Wentong
Quoted 71 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Tuesday, March 7, 2023 4:30 PMHi Wentong,
Quoted 27 lines · Wentong
Quoted 27 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Wednesday, March 1, 2023 6:42 PMHi,
Quoted 18 lines · Sakari Ailus
Hi Wentong,
Quoted 15 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking
up systemQuoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided
by this IVSC driver.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware
Quoted 8 lines
Quoted 8 lines
modules are:
Quoted 4 lines
Quoted 4 lines
- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownershipof camera sensor.
Quoted 2 lines
Quoted 2 lines
- CSI (Camera Serial Interface): This module is used to route
camera sensor data either to IVSC or to host for IPU driver andapplication.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be
notified.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
I thought this for a while and did some research, and I can suggest the
following:- The IVSC sub-device implements a control for privacy
(
V4L2_CID_PRIVACYQuoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
is a good fit).
- Camera sensor access needs to be requested from IVSC before accessing
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
the
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
sensor via I²C. The IVSC ownership control needs to be in the right setting for this to work, and device links can be used for that purpose (see
device_link_add()). WithDL_FLAG_PM_RUNTIMEand
DL_FLAG_RPM_ACTIVE,Quoted 2 lines
the supplier devices will be PM runtime resumed before the consumer (camera sensor). As these devices are purely virtual on host side and
has
Quoted 36 lines
Quoted 36 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
no power state as such, you can use runtime PM callbacks to
transferthe
Quoted 20 lines
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
Quoted 1 line
ownership.
Interesting proposal to use device-links + runtime-pm for this instead of modelling this as an i2c-mux. FWIW I'm fine with going this route instead of using an i2c-mux approach.
I have been thinking about the i2c-mux approach a bit and the problem is that we are not really muxing but want to turn on/off control and AFAIK the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain, so control will never be released when the i2c
transfers are done.
Quoted 6 lines
Quoted 6 lines
And if were to somehow modify things (or maybe there already is some release
callback) then the downside becomes that the i2c-mux core code operates at the i2c transfer level. So each i2c read/write would then enable +disavle control.
Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Modelling this using something like runtime pm as such is a much better fit because then we request control once on probe / stream-on and release it once we are fully done, rather then requesting + releasing control once per i2c- transfer.
Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe, probably we have to switch to i2c-mux modeling
way.
Quoted 4 lines
Quoted 3 lines
What do you mean? The supplier devices are resumed before the driver's probe is called.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
But we setup the link with
device_link_addduring IVSC driver's probe, we can't guarantee driver probe's sequence.Then maybe we need to do the
device_link_addsomewhere else.sensor's parent is the LJCA I2C device whose driver is being upstream
https://www.spinics.net/lists/kernel/msg4702552.htmland and sensor's power is controlled by IVSC instead of INT3472 if IVSC enabled.
Quoted 6 lines
I believe that the INT3472 code is still involved at least on a Dell Latitude 9420 the INT3472 code still needs to set the clock-enable and the privacy-LED GPIOs otherwise the main camera won't work.
So I'm not sure what you mean with "sensor's power is controlled by IVSC instead of INT3472" ?
Could you please share your kernel log and DSDT? Thanks
BR,
Wentong
Quoted 91 lines
Quoted 14 lines
struct
device_link*device_link_add(struct device *consumer,
struct device *supplier, u32
flags)So probably we have to add above
device_link_addin LJCA I2C's driver, and we can find the consumer(camera sensor) with ACPI API, but the supplier,mei_ace, is mei client device under mei framework and it's dynamically allocated device instead of ACPI device, probably I can find its parent with some ACPI lookup from this LJCA I2C device, but unfortunately mei framework doesn't export the API to find mei client device with its parent busdevice(struct mei_device).Quoted 3 lines
I'm not sure if modeling this
mei_aceas LJCA I2C's runtime power control is acceptable, if yes, probably thismei_acedriver have to go with LJCA I2C device driver.Looking at the ACPI table the sensor ACPI device has 2 _DEP-s listed the I2C controller and the INT3472 device. Since we are already doing similar setup in the INT3472 device that seems like a good place to add the device_link()-s (it can return -EPROBE_DEFER to wait for the
mei_aceto show up).But when the INT3472 code runs, the consumer device does not exist yet and AFAICT the same is true when the LCJA i2c-controller driver is getting registered. The consumer only exists when the
i2c_clientis instantiated and at that point the sensor driversprobe()method can run immediately and we are too late to add thedevice_link.As a hobby project I have been working on atomisp2 support and I have a similar issue there. There is no INT3472 device there, but there is a
_DSMmethod which needs to be used to figure out which ACPI GPIO resource is reset / powerdown and if the GPIOs are active-low or active high.I have written a little helper function to call the
_DSMand to then turn this into lookups and calldevm_acpi_dev_add_driver_gpios().Since on atomisp2 we cannot use the INT3472 driver to delay the sensor-driver probe and have the INT3472 driver setup the GPIO lookup, at least for the sensor drivers used with
atomisp2 there is going to be a need to add a single line toprobe()like this:v4l2_get_acpi_sensor_info(&i2c_client->dev, NULL);To me it sounds like we need to do something similar here and extend the helper function which I have written (but not yet submitted upstream) :
https://github.com/jwrdegoede/linux-
sunxi/commit/e2287979db43d46fa7d354c1bde92eb6219b613dTo also setup the device-links needed for the runtime-pm solution to getting the i2c passed through to the sensor.
Ideally
v4l2_get_acpi_sensor_info()should return void (easier to use in the sensor drivers) but I think it should return an int, so that it can e.g. return -EPROBE_DEFERto wait for themei_ace.Regards,
Hans
Quoted 10 lines
Quoted 10 lines
The mainline kernel delays probing of camera sensors on Intel platforms until the INT3472 driver has probed the INT3472 device on which the sensors have an ACPI
_DEP.This is already used to make sure that clock lookups and regulator info is in place before the sensor's
probe()function runs.So that when the driver does
clk_get()it succeeds and so thatregulator_get()does not end up returning a dummy regulator.Quoted 7 lines
Quoted 7 lines
So I think the code adding the device_link-s for the IVSC should be added
to:drivers/platform/x86/intel/int3472/discrete.cand then the runtime-resume will happen before the sensor'sprobe()function runs.Likewise
drivers/platform/x86/intel/int3472/discrete.cshould also ensure that the ivsc driver'sprobe()has run before it calls
acpi_dev_clear_dependencies().Quoted 9 lines
Quoted 8 lines
The
acpi_dev_clear_dependencies()call indiscrete.ctells the ACPI subsystem to go ahead and create the i2c-clients for the sensors and allow the sensor drivers to get loaded and probe the sensor.Regards,
Hans
<re-added the previous Cc list, which I dropped because of the large attachments>
Hi Wentong,
Quoted 6 lines · Wentong
Hi Hans,
Thanks
And AFAICT, there is no IVSC device on your Dell Latitude 9420 where the platform is based on TGL instead of ADL, and I have never heard IVSC runs on TGL, if no IVSC, INT3472 will control sensor's power. And I will double confirm with people who know dell product well tomorrow.
Ah, I was under the impression that there was an IVSC there because:
1. The sensor driver for the used sensor (tries to) poke the IVSC
2. Things did not work without building the IVSC drivers, but that might be due to a dependency on the LCJA GPIO expander instead
But you might very well be right, that would also explain the "intel vsc not ready" messages in dmesg.
If with the IVSC case the IVSC controls the power to the sensor too, then another option might be to model the I2C-switch + the power-control as a powerdown GPIO for the sensor, which most sensor drivers already try to use. The advantage of doing this would be that GPIO lookups can reference the GPIO provider + consumer by device-name so then we don't need to have both devices instantiated at the time of
adding the GPIO lookup. And in that case we could e.g. add the lookup
before registering the I2C controller.
Sakari, what do you think of instead of using runtime-pm + devlinks having the IVSC code export a powerdown GPIO to the sensor ?
This also decouples the ivsc powerstate from the sensor power-state which might be useful if we ever want to use some of the more advanced ivsc features, where AFAICT the ivsc fully controls the sensor.
Regards,
Hans
Quoted 282 lines
Quoted 282 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 9:30 PM
To: Wu, Wentong <[email protected]>
Subject: Re: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)Hi Wentong,
Attached are the requested dmesg + acpidump for the Dell Latitude 9420.
Regards,
Hans
Quoted 49 lines · Wentong
Hi
Quoted 47 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 5:28 PMHi,
Quoted 40 lines · Wentong
Quoted 40 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Tuesday, March 7, 2023 5:10 PMHi,
Quoted 31 lines · Wentong
Quoted 31 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Tuesday, March 7, 2023 4:30 PMHi Wentong,
Quoted 22 lines · Wentong
Quoted 22 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Wednesday, March 1, 2023 6:42 PMHi,
Quoted 13 lines · Sakari Ailus
Hi Wentong,
Quoted 10 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple
sensors.
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking
up systemQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the
CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces providedby this IVSC driver.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware
modules are:
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
- ACE (Algorithm Context Engine): This module is for
algorithm computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownershipof camera sensor.
Quoted 3 lines
Quoted 3 lines
- CSI (Camera Serial Interface): This module is used to
route camera sensor data either to IVSC or to host for IPU driver andapplication.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be
notified.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
I thought this for a while and did some research, and I can suggest the
following:- The IVSC sub-device implements a control for privacy
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
(
V4L2_CID_PRIVACYQuoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
is a good fit).
- Camera sensor access needs to be requested from IVSC before accessing
the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
sensor via I²C. The IVSC ownership control needs to be in the right setting for this to work, and device links can be used for that
purpose
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
(see
device_link_add()). WithDL_FLAG_PM_RUNTIMEand
DL_FLAG_RPM_ACTIVE,Quoted 1 line
the supplier devices will be PM runtime resumed before the
consumer
Quoted 52 lines
Quoted 52 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(camera sensor). As these devices are purely virtual on host
side andhas
Quoted 36 lines
Quoted 36 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
no power state as such, you can use runtime PM callbacks to
transferthe
Quoted 20 lines
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
Quoted 1 line
ownership.
Interesting proposal to use device-links + runtime-pm for this instead of modelling this as an i2c-mux. FWIW I'm fine with going this route instead of using an i2c-mux approach.
I have been thinking about the i2c-mux approach a bit and the problem is that we are not really muxing but want to turn on/off control and AFAIK the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain, so control will never be released when the i2c
transfers are done.
Quoted 6 lines
Quoted 6 lines
And if were to somehow modify things (or maybe there already is some release
callback) then the downside becomes that the i2c-mux core code operates at the i2c transfer level. So each i2c read/write would then enable +disavle control.
Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Modelling this using something like runtime pm as such is a much better fit because then we request control once on probe / stream-on and release it once we are fully done, rather then requesting + releasing control once per i2c- transfer.
Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe, probably we have to switch to i2c-mux modeling
way.
Quoted 4 lines
Quoted 3 lines
What do you mean? The supplier devices are resumed before the driver's probe is called.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
But we setup the link with
device_link_addduring IVSC driver's probe, we can't guarantee driver probe's sequence.Then maybe we need to do the
device_link_addsomewhere else.sensor's parent is the LJCA I2C device whose driver is being upstream https://www.spinics.net/lists/kernel/msg4702552.htmland and sensor's power is controlled by IVSC instead of INT3472 if IVSC enabled.
I believe that the INT3472 code is still involved at least on a Dell Latitude 9420 the INT3472 code still needs to set the clock-enable and the privacy-LED GPIOs otherwise the main camera won't work.
Quoted 29 lines
Quoted 2 lines
So I'm not sure what you mean with "sensor's power is controlled by IVSC instead of INT3472" ?
Could you please share your kernel log and DSDT? Thanks
BR,
WentongQuoted 22 lines
Quoted 14 lines
struct
device_link*device_link_add(struct device *consumer,
struct device *supplier, u32
flags)So probably we have to add above
device_link_addin LJCA I2C's driver, and we can find the consumer(camera sensor) with ACPI API, but the supplier,mei_ace, is mei client device under mei framework and it's dynamically allocated device instead of ACPI device, probably I can find its parent with some ACPI lookup from this LJCA I2C device, but unfortunately mei framework doesn't export the API to find mei client device with its parent busdevice(struct mei_device).Quoted 3 lines
I'm not sure if modeling this
mei_aceas LJCA I2C's runtime power control is acceptable, if yes, probably thismei_acedriver have to go with LJCA I2C device driver.Looking at the ACPI table the sensor ACPI device has 2 _DEP-s listed the I2C controller and the INT3472 device. Since we are already doing similar setup in the INT3472 device that seems like a good place to add the device_link()-s (it can return -EPROBE_DEFER to wait for the
mei_aceto show up).
Quoted 3 lines
Quoted 3 lines
But when the INT3472 code runs, the consumer device does not exist yet and AFAICT the same is true when the LCJA i2c-controller driver is getting
registered.
Quoted 67 lines
Quoted 66 lines
The consumer only exists when the
i2c_clientis instantiated and at that point the sensor driversprobe()method can run immediately and we are too late to add thedevice_link.As a hobby project I have been working on atomisp2 support and I have a similar issue there. There is no INT3472 device there, but there is a
_DSMmethod which needs to be used to figure out which ACPI GPIO resource is reset / powerdown and if the GPIOs are active-low or active high.I have written a little helper function to call the
_DSMand to then turn this into lookups and calldevm_acpi_dev_add_driver_gpios().Since on atomisp2 we cannot use the INT3472 driver to delay the sensor-driver probe and have the INT3472 driver setup the GPIO lookup, at least for the sensor drivers used with
atomisp2 there is going to be a need to add a single line toprobe()like this:v4l2_get_acpi_sensor_info(&i2c_client->dev, NULL);To me it sounds like we need to do something similar here and extend the helper function which I have written (but not yet submitted upstream) :
https://github.com/jwrdegoede/linux-
sunxi/commit/e2287979db43d46fa7d354c1bde92eb6219b613dTo also setup the device-links needed for the runtime-pm solution to getting the i2c passed through to the sensor.
Ideally
v4l2_get_acpi_sensor_info()should return void (easier to use in the sensor drivers) but I think it should return an int, so that it can e.g. return -EPROBE_DEFERto wait for themei_ace.Regards,
Hans
Quoted 10 lines
Quoted 10 lines
The mainline kernel delays probing of camera sensors on Intel platforms until the INT3472 driver has probed the INT3472 device on which the sensors have an ACPI
_DEP.This is already used to make sure that clock lookups and regulator info is in place before the sensor's
probe()function runs.So that when the driver does
clk_get()it succeeds and so thatregulator_get()does not end up returning a dummy regulator.Quoted 7 lines
Quoted 7 lines
So I think the code adding the device_link-s for the IVSC should be added
to:drivers/platform/x86/intel/int3472/discrete.cand then the runtime-resume will happen before the sensor'sprobe()function runs.Likewise
drivers/platform/x86/intel/int3472/discrete.cshould also ensure that the ivsc driver'sprobe()has run before it calls
acpi_dev_clear_dependencies().Quoted 9 lines
Quoted 8 lines
The
acpi_dev_clear_dependencies()call indiscrete.ctells the ACPI subsystem to go ahead and create the i2c-clients for the sensors and allow the sensor drivers to get loaded and probe the sensor.Regards,
Hans
Quoted 19 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 11:24 PM<re-added the previous Cc list, which I dropped because of the large attachments>
Hi Wentong,
Quoted 5 lines · Wentong
Hi Hans,
Thanks
And AFAICT, there is no IVSC device on your Dell Latitude 9420 where the
platform is based on TGL instead of ADL, and I have never heard IVSC runs on TGL, if no IVSC, INT3472 will control sensor's power.
Quoted 7 lines
Quoted 1 line
And I will double confirm with people who know dell product well tomorrow.
Ah, I was under the impression that there was an IVSC there because:
1. The sensor driver for the used sensor (tries to) poke the IVSC 2. Things did not work without building the IVSC drivers, but that might
be due to a dependency on the LCJA GPIO expander instead
Below is your dmesg log, the required SPI controller for IVSC isn't here.
[ 35.538114] ljca 2-6:1.0: acked sem wait timed out ret:0 timeout:20 ack:0 [ 35.538129] ljca 2-6:1.0: MNG_ENUM_SPI failed ret:-110 len:7 num:0 [ 35.538621] ljca 2-6:1.0: LJCA USB device init success [ 35.538776] usbcore: registered new interface driver ljca
Also I checked your SSDT, there is no IVSC device and the sensor device depends on INT3472 instead of IVSC device as on my setup.
Quoted 4 lines
But you might very well be right, that would also explain the "intel vsc not ready" messages in dmesg.
Quoted 8 lines
If with the IVSC case the IVSC controls the power to the sensor too, then another option might be to model the I2C-switch + the power-control as a powerdown GPIO for the sensor, which most sensor drivers already try to use. The advantage of doing this would be that GPIO lookups can reference the GPIO provider + consumer by device-name so then we don't need to have both devices instantiated at the time of
adding the GPIO lookup. And in that case we could e.g. add the lookup
before registering the I2C controller.
Can we add IVSC device to acpi_honor_dep_ids, so that when everything is done during mei_ace probe, acpi_dev_clear_dependencies can make sensor start probe?
BR,
Wentong
Quoted 309 lines
Sakari, what do you think of instead of using runtime-pm + devlinks having the IVSC code export a powerdown GPIO to the sensor ?
This also decouples the ivsc powerstate from the sensor power-state which might be useful if we ever want to use some of the more advanced ivsc features, where AFAICT the ivsc fully controls the sensor.
Regards,
Hans
Quoted 83 lines
Quoted 83 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 9:30 PM
To: Wu, Wentong <[email protected]>
Subject: Re: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)Hi Wentong,
Attached are the requested dmesg + acpidump for the Dell Latitude 9420.
Regards,
Hans
Quoted 50 lines · Wentong
Hi
Quoted 48 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 5:28 PMHi,
Quoted 41 lines · Wentong
Quoted 41 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Tuesday, March 7, 2023 5:10 PMHi,
Quoted 32 lines · Wentong
Quoted 32 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Tuesday, March 7, 2023 4:30 PMHi Wentong,
Quoted 23 lines · Wentong
Quoted 23 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Wednesday, March 1, 2023 6:42 PMHi,
Quoted 14 lines · Sakari Ailus
Hi Wentong,
Quoted 11 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover
Falls", is a companion chip designed to provide secure and
low power vision capability to IA platforms. IVSC is
available in existing commercial platforms from multiple OEMs.The primary use case of IVSC is to bring in context awareness.
IVSC interfaces directly with the platform main camera
sensor via a CSI-2 link and processes the image data with
the embedded AI engine. The detected events are sent over
I2C to ISH (Intel Sensor Hub) for additional data fusion
from multiplesensors.
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or
waking up systemSince the Image Processing Unit(IPU) used on the host
processor needs to configure the CSI-2 link in normal
camera usages, the
CSI-2 link and camera sensor can only be used in
mutually-exclusive ways by host IPU and IVSC. By default
the IVSC owns the CSI-2 link and camera sensor. The IPU
driver can take ownership of the CSI-2 link and camera
sensor using interfaces providedQuoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
by this IVSC driver.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Switching ownership requires an interface with two
different hardware modules inside IVSC. The software
interface to these modules is via Intel MEI (The IntelManagement Engine) commands.
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines
Quoted 2 lines
These two hardware modules have two different MEI UUIDs to
enumerate. These hardwaremodules are:
Quoted 4 lines
Quoted 4 lines
- ACE (Algorithm Context Engine): This module is for
algorithm computing when IVSC owns camera sensor. Also ACE
module controls camera sensor's ownership. This hardware
module is used to set ownershipof camera sensor.
Quoted 3 lines
Quoted 3 lines
- CSI (Camera Serial Interface): This module is used to
route camera sensor data either to IVSC or to host for IPU
driver andapplication.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
IVSC also provides a privacy mode. When privacy mode is
turned on, camera sensor can't be used. This means that
both ACE and host IPU can't get image data. And when this
mode is turned on, host IPU driver is informed via a
registered callback, so that user can benotified.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 4 lines
In summary, to acquire ownership of camera by IPU driver,
first ACE module needs to be informed of ownership and then
to setup MIPI CSI-2 link for the camera sensor and IPU.I thought this for a while and did some research, and I can suggest the
following:Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
- The IVSC sub-device implements a control for privacy
(
V4L2_CID_PRIVACYQuoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
is a good fit).
- Camera sensor access needs to be requested from IVSC before accessing
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
sensor via I²C. The IVSC ownership control needs to be in the
right
Quoted 46 lines
Quoted 46 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
setting for this to work, and device links can be used for
thatpurpose
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
(see
device_link_add()). WithDL_FLAG_PM_RUNTIMEand
DL_FLAG_RPM_ACTIVE,Quoted 1 line
the supplier devices will be PM runtime resumed before the
consumer
Quoted 39 lines
Quoted 39 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(camera sensor). As these devices are purely virtual on
host side andhas
Quoted 36 lines
Quoted 36 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
no power state as such, you can use runtime PM callbacks
to transferthe
Quoted 20 lines
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
Quoted 1 line
ownership.
Interesting proposal to use device-links + runtime-pm for this instead of modelling this as an i2c-mux. FWIW I'm fine with going this route instead of using an i2c-mux approach.
I have been thinking about the i2c-mux approach a bit and the problem is that we are not really muxing but want to turn on/off control and AFAIK the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain, so control will never be released when the i2c
transfers are done.
Quoted 6 lines
Quoted 6 lines
And if were to somehow modify things (or maybe there already is some release
callback) then the downside becomes that the i2c-mux core code operates at the i2c transfer level. So each i2c read/write would then enable +disavle control.
Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Modelling this using something like runtime pm as such is a much better fit because then we request control once on probe / stream-on and release it once we are fully done, rather then requesting + releasing control once per i2c- transfer.
Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe, probably we have to switch to i2c-mux modeling
way.
Quoted 4 lines
Quoted 3 lines
What do you mean? The supplier devices are resumed before the driver's probe is called.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
But we setup the link with
device_link_addduring IVSC driver's probe, we can't guarantee driver probe's sequence.Then maybe we need to do the
device_link_addsomewhere else.sensor's parent is the LJCA I2C device whose driver is being upstream https://www.spinics.net/lists/kernel/msg4702552.htmland
and sensor's power is controlled by IVSC instead of INT3472 if IVSCenabled.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
I believe that the INT3472 code is still involved at least on a Dell Latitude 9420 the INT3472 code still needs to set the clock-enable and the privacy-LED GPIOs otherwise the main camera won't
work.
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 3 lines
So I'm not sure what you mean with "sensor's power is controlled by IVSC instead of INT3472" ?
Could you please share your kernel log and DSDT? Thanks
BR,
WentongQuoted 6 lines
Quoted 6 lines
struct
device_link*device_link_add(struct device *consumer,
struct device *supplier, u32
flags)Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
So probably we have to add above
device_link_addin LJCA I2C's driver, and we can find the consumer(camera sensor) with ACPI API, but the supplier,mei_ace, is mei client device under mei framework and it's dynamically allocated device instead of ACPI device, probably I can find its parent with some ACPI lookup from this LJCA I2C device, but unfortunately mei framework doesn't export the API to find mei client device with its parent bus device(struct
mei_device).Quoted 16 lines
Quoted 16 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
I'm not sure if modeling this
mei_aceas LJCA I2C's runtime power control is acceptable, if yes, probably thismei_acedriver have to go with LJCA I2C device driver.Looking at the ACPI table the sensor ACPI device has 2 _DEP-s listed the I2C controller and the INT3472 device. Since we are already doing similar setup in the INT3472 device that seems like a good place to add the device_link()-s (it can return -EPROBE_DEFER to wait for the
mei_aceto show up).
Quoted 4 lines
Quoted 4 lines
But when the INT3472 code runs, the consumer device does not exist yet and AFAICT the same is true when the LCJA i2c-controller driver is getting
registered.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
The consumer only exists when the
i2c_clientis instantiated and at that point the sensor driversprobe()method can run immediately and we are too late to add thedevice_link.As a hobby project I have been working on atomisp2 support and I have a similar issue there. There is no INT3472 device there, but there is a
_DSMmethod which needs to be used to figure out which ACPI GPIO resource is reset / powerdown and if the GPIOs are active-lowor active high.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
I have written a little helper function to call the
_DSMand to then turn this into lookups and calldevm_acpi_dev_add_driver_gpios().Since on atomisp2 we cannot use the INT3472 driver to delay the sensor-driver probe and have the INT3472 driver setup the GPIO lookup, at least for the sensor drivers used with
atomisp2 there is going to be a need to add a single line toprobe()like this:Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
v4l2_get_acpi_sensor_info(&i2c_client->dev, NULL);To me it sounds like we need to do something similar here and extend the helper function which I have written (but not yet submitted
upstream) :
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 46 lines
https://github.com/jwrdegoede/linux-
sunxi/commit/e2287979db43d46fa7d354c1bde92eb6219b613dTo also setup the device-links needed for the runtime-pm solution to getting the i2c passed through to the sensor.
Ideally
v4l2_get_acpi_sensor_info()should return void (easier to use in the sensor drivers) but I think it should return an int, so that it can e.g. return -EPROBE_DEFERto wait for themei_ace.Regards,
Hans
Quoted 10 lines
Quoted 10 lines
The mainline kernel delays probing of camera sensors on Intel platforms until the INT3472 driver has probed the INT3472 device on which the sensors have an ACPI
_DEP.This is already used to make sure that clock lookups and regulator info is in place before the sensor's
probe()function runs.So that when the driver does
clk_get()it succeeds and so thatregulator_get()does not end up returning a dummy regulator.Quoted 8 lines
Quoted 8 lines
So I think the code adding the device_link-s for the IVSC should be added
to:drivers/platform/x86/intel/int3472/discrete.cand then the runtime-resume will happen before the sensor'sprobe()function runs.Likewise
drivers/platform/x86/intel/int3472/discrete.cshould also ensure that the ivsc driver'sprobe()has run before it calls
acpi_dev_clear_dependencies().Quoted 9 lines
Quoted 8 lines
The
acpi_dev_clear_dependencies()call indiscrete.ctells the ACPI subsystem to go ahead and create the i2c-clients for the sensors and allow the sensor drivers to get loaded and probe the sensor.Regards,
Hans
Hi,
Quoted 29 lines · Wentong
Quoted 19 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 11:24 PM<re-added the previous Cc list, which I dropped because of the large attachments>
Hi Wentong,
Quoted 5 lines · Wentong
Hi Hans,
Thanks
And AFAICT, there is no IVSC device on your Dell Latitude 9420 where the
platform is based on TGL instead of ADL, and I have never heard IVSC runs on TGL, if no IVSC, INT3472 will control sensor's power.
Quoted 7 lines
Quoted 1 line
And I will double confirm with people who know dell product well tomorrow.
Ah, I was under the impression that there was an IVSC there because:
1. The sensor driver for the used sensor (tries to) poke the IVSC 2. Things did not work without building the IVSC drivers, but that might
be due to a dependency on the LCJA GPIO expander insteadBelow is your dmesg log, the required SPI controller for IVSC isn't here.
Quoted 7 lines
[ 35.538114] ljca 2-6:1.0: acked sem wait timed out ret:0 timeout:20 ack:0 [ 35.538129] ljca 2-6:1.0: MNG_ENUM_SPI failed ret:-110 len:7 num:0 [ 35.538621] ljca 2-6:1.0: LJCA USB device init success [ 35.538776] usbcore: registered new interface driver ljcaAlso I checked your SSDT, there is no IVSC device and the sensor device depends on INT3472 instead of IVSC device as on my setup.
Ack.
Quoted 12 lines
Quoted 11 lines
But you might very well be right, that would also explain the "intel vsc not ready" messages in dmesg.
If with the IVSC case the IVSC controls the power to the sensor too, then another option might be to model the I2C-switch + the power-control as a powerdown GPIO for the sensor, which most sensor drivers already try to use. The advantage of doing this would be that GPIO lookups can reference the GPIO provider + consumer by device-name so then we don't need to have both devices instantiated at the time of
adding the GPIO lookup. And in that case we could e.g. add the lookup
before registering the I2C controller.
Quoted 2 lines
Can we add IVSC device to
acpi_honor_dep_ids, so that when everything is done duringmei_aceprobe,acpi_dev_clear_dependenciescan make sensor start probe?
Does the sensor ACPI device node have an ACPI _DEP on the IVSC device ?
If yes, then yes we can add the IVSC device to acpi_honor_dep_id and make mei_ace probe call acpi_dev_clear_dependencies().
Regards,
Hans
Quoted 309 lines
Quoted 308 lines
Sakari, what do you think of instead of using runtime-pm + devlinks having the IVSC code export a powerdown GPIO to the sensor ?
This also decouples the ivsc powerstate from the sensor power-state which might be useful if we ever want to use some of the more advanced ivsc features, where AFAICT the ivsc fully controls the sensor.
Regards,
Hans
Quoted 83 lines
Quoted 83 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 9:30 PM
To: Wu, Wentong <[email protected]>
Subject: Re: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)Hi Wentong,
Attached are the requested dmesg + acpidump for the Dell Latitude 9420.
Regards,
Hans
Quoted 50 lines · Wentong
Hi
Quoted 48 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 5:28 PMHi,
Quoted 41 lines · Wentong
Quoted 41 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Tuesday, March 7, 2023 5:10 PMHi,
Quoted 32 lines · Wentong
Quoted 32 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Tuesday, March 7, 2023 4:30 PMHi Wentong,
Quoted 23 lines · Wentong
Quoted 23 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Wednesday, March 1, 2023 6:42 PMHi,
Quoted 14 lines · Sakari Ailus
Hi Wentong,
On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
Intel Visual Sensing Controller (IVSC), codenamed "Clover
Falls", is a companion chip designed to provide secure and
low power vision capability to IA platforms. IVSC is
available in existing commercial platforms from multiple OEMs.The primary use case of IVSC is to bring in context awareness.
IVSC interfaces directly with the platform main camera
sensor via a CSI-2 link and processes the image data with
the embedded AI engine. The detected events are sent over
I2C to ISH (Intel Sensor Hub) for additional data fusion
from multiplesensors.
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or
waking up systemSince the Image Processing Unit(IPU) used on the host
processor needs to configure the CSI-2 link in normal
camera usages, the
CSI-2 link and camera sensor can only be used in
mutually-exclusive ways by host IPU and IVSC. By default
the IVSC owns the CSI-2 link and camera sensor. The IPU
driver can take ownership of the CSI-2 link and camera
sensor using interfaces providedQuoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
by this IVSC driver.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Switching ownership requires an interface with two
different hardware modules inside IVSC. The software
interface to these modules is via Intel MEI (The IntelManagement Engine) commands.
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines
These two hardware modules have two different MEI UUIDs to
enumerate. These hardwaremodules are:
Quoted 4 lines
- ACE (Algorithm Context Engine): This module is for
algorithm computing when IVSC owns camera sensor. Also ACE
module controls camera sensor's ownership. This hardware
module is used to set ownershipof camera sensor.
Quoted 3 lines
- CSI (Camera Serial Interface): This module is used to
route camera sensor data either to IVSC or to host for IPU
driver andapplication.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
IVSC also provides a privacy mode. When privacy mode is
turned on, camera sensor can't be used. This means that
both ACE and host IPU can't get image data. And when this
mode is turned on, host IPU driver is informed via a
registered callback, so that user can benotified.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
In summary, to acquire ownership of camera by IPU driver,
first ACE module needs to be informed of ownership and then
to setup MIPI CSI-2 link for the camera sensor and IPU.I thought this for a while and did some research, and I can
suggest the
following:Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
- The IVSC sub-device implements a control for privacy
(
V4L2_CID_PRIVACYQuoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
is a good fit).
- Camera sensor access needs to be requested from IVSC
before accessingthe
Quoted 1 line
Quoted 1 line
Quoted 1 line
sensor via I²C. The IVSC ownership control needs to be in the
right
Quoted 46 lines
Quoted 46 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
setting for this to work, and device links can be used for
thatpurpose
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
(see device_link_add()). With DL_FLAG_PM_RUNTIME and
DL_FLAG_RPM_ACTIVE,Quoted 1 line
the supplier devices will be PM runtime resumed before the
consumer
Quoted 39 lines
Quoted 39 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(camera sensor). As these devices are purely virtual on
host side andhas
Quoted 36 lines
Quoted 36 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
no power state as such, you can use runtime PM callbacks
to transferthe
Quoted 20 lines
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
Quoted 1 line
ownership.
Interesting proposal to use device-links + runtime-pm for this instead of modelling this as an i2c-mux. FWIW I'm fine with going this route instead of using an i2c-mux approach.
I have been thinking about the i2c-mux approach a bit and the problem is that we are not really muxing but want to turn on/off control and AFAIK the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain, so control will never be released when the i2c
transfers are done.
Quoted 6 lines
Quoted 6 lines
And if were to somehow modify things (or maybe there already is some release
callback) then the downside becomes that the i2c-mux core code operates at the i2c transfer level. So each i2c read/write would then enable +disavle control.
Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Modelling this using something like runtime pm as such is a much better fit because then we request control once on probe / stream-on and release it once we are fully done, rather then requesting + releasing control once per i2c- transfer.
Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe, probably we have to switch to i2c-mux modeling
way.
Quoted 4 lines
Quoted 3 lines
What do you mean? The supplier devices are resumed before the driver's probe is called.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
But we setup the link with
device_link_addduring IVSC driver's probe, we can't guarantee driver probe's sequence.Then maybe we need to do the
device_link_addsomewhere else.sensor's parent is the LJCA I2C device whose driver is being upstream https://www.spinics.net/lists/kernel/msg4702552.htmland
and sensor's power is controlled by IVSC instead of INT3472 if IVSCenabled.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
I believe that the INT3472 code is still involved at least on a Dell Latitude 9420 the INT3472 code still needs to set the clock-enable and the privacy-LED GPIOs otherwise the main camera won't
work.
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 3 lines
So I'm not sure what you mean with "sensor's power is controlled by IVSC instead of INT3472" ?
Could you please share your kernel log and DSDT? Thanks
BR,
WentongQuoted 6 lines
Quoted 6 lines
struct
device_link*device_link_add(struct device *consumer,
struct device *supplier, u32
flags)Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
So probably we have to add above
device_link_addin LJCA I2C's driver, and we can find the consumer(camera sensor) with ACPI API, but the supplier,mei_ace, is mei client device under mei framework and it's dynamically allocated device instead of ACPI device, probably I can find its parent with some ACPI lookup from this LJCA I2C device, but unfortunately mei framework doesn't export the API to find mei client device with its parent bus device(struct
mei_device).Quoted 16 lines
Quoted 16 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
I'm not sure if modeling this
mei_aceas LJCA I2C's runtime power control is acceptable, if yes, probably thismei_acedriver have to go with LJCA I2C device driver.Looking at the ACPI table the sensor ACPI device has 2 _DEP-s listed the I2C controller and the INT3472 device. Since we are already doing similar setup in the INT3472 device that seems like a good place to add the device_link()-s (it can return -EPROBE_DEFER to wait for the
mei_aceto show up).
Quoted 4 lines
Quoted 4 lines
But when the INT3472 code runs, the consumer device does not exist yet and AFAICT the same is true when the LCJA i2c-controller driver is getting
registered.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
The consumer only exists when the
i2c_clientis instantiated and at that point the sensor driversprobe()method can run immediately and we are too late to add thedevice_link.As a hobby project I have been working on atomisp2 support and I have a similar issue there. There is no INT3472 device there, but there is a
_DSMmethod which needs to be used to figure out which ACPI GPIO resource is reset / powerdown and if the GPIOs are active-lowor active high.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
I have written a little helper function to call the
_DSMand to then turn this into lookups and calldevm_acpi_dev_add_driver_gpios().Since on atomisp2 we cannot use the INT3472 driver to delay the sensor-driver probe and have the INT3472 driver setup the GPIO lookup, at least for the sensor drivers used with
atomisp2 there is going to be a need to add a single line toprobe()like this:Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
v4l2_get_acpi_sensor_info(&i2c_client->dev, NULL);To me it sounds like we need to do something similar here and extend the helper function which I have written (but not yet submitted
upstream) :
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 46 lines
https://github.com/jwrdegoede/linux-
sunxi/commit/e2287979db43d46fa7d354c1bde92eb6219b613dTo also setup the device-links needed for the runtime-pm solution to getting the i2c passed through to the sensor.
Ideally
v4l2_get_acpi_sensor_info()should return void (easier to use in the sensor drivers) but I think it should return an int, so that it can e.g. return -EPROBE_DEFERto wait for themei_ace.Regards,
Hans
Quoted 10 lines
Quoted 10 lines
The mainline kernel delays probing of camera sensors on Intel platforms until the INT3472 driver has probed the INT3472 device on which the sensors have an ACPI
_DEP.This is already used to make sure that clock lookups and regulator info is in place before the sensor's
probe()function runs.So that when the driver does
clk_get()it succeeds and so thatregulator_get()does not end up returning a dummy regulator.Quoted 8 lines
Quoted 8 lines
So I think the code adding the device_link-s for the IVSC should be added
to:drivers/platform/x86/intel/int3472/discrete.cand then the runtime-resume will happen before the sensor'sprobe()function runs.Likewise
drivers/platform/x86/intel/int3472/discrete.cshould also ensure that the ivsc driver'sprobe()has run before it calls
acpi_dev_clear_dependencies().Quoted 9 lines
Quoted 8 lines
The
acpi_dev_clear_dependencies()call indiscrete.ctells the ACPI subsystem to go ahead and create the i2c-clients for the sensors and allow the sensor drivers to get loaded and probe the sensor.Regards,
Hans
Quoted 64 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 16, 2023 5:04 PMHi,
Quoted 30 lines · Wentong
Quoted 20 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 11:24 PM<re-added the previous Cc list, which I dropped because of the large attachments>
Hi Wentong,
Quoted 6 lines · Wentong
Hi Hans,
Thanks
And AFAICT, there is no IVSC device on your Dell Latitude 9420 where the
platform is based on TGL instead of ADL, and I have never heard IVSC runs on TGL, if no IVSC, INT3472 will control sensor's power.
Quoted 7 lines
Quoted 1 line
And I will double confirm with people who know dell product well tomorrow.
Ah, I was under the impression that there was an IVSC there because:
1. The sensor driver for the used sensor (tries to) poke the IVSC 2. Things did not work without building the IVSC drivers, but that might be due to a dependency on the LCJA GPIO expander instead
Below is your dmesg log, the required SPI controller for IVSC isn't here.
Quoted 8 lines
[ 35.538114] ljca 2-6:1.0: acked sem wait timed out ret:0 timeout:20 ack:0 [ 35.538129] ljca 2-6:1.0: MNG_ENUM_SPI failed ret:-110 len:7 num:0 [ 35.538621] ljca 2-6:1.0: LJCA USB device init success [ 35.538776] usbcore: registered new interface driver ljcaAlso I checked your SSDT, there is no IVSC device and the sensor device depends on
INT3472 instead of IVSC device as on my setup.Ack.
Quoted 1 line
Quoted 1 line
But you might very well be right, that would also explain the "intel vsc not
ready"
Quoted 5 lines
Quoted 5 lines
messages in dmesg.
If with the IVSC case the IVSC controls the power to the sensor too, then another option might be to model the I2C-switch + the power-control as a powerdown GPIO for the sensor, which most sensor
drivers already try to use.
Quoted 6 lines
Quoted 5 lines
The advantage of doing this would be that GPIO lookups can reference the GPIO provider + consumer by device-name so then we don't need to have both devices instantiated at the time of
adding the GPIO lookup. And in that case we could e.g. add the lookup
before registering the I2C controller.
Quoted 5 lines
Quoted 2 lines
Can we add IVSC device to
acpi_honor_dep_ids, so that when everything is done duringmei_aceprobe,acpi_dev_clear_dependenciescan make sensorstart probe?
Does the sensor ACPI device node have an ACPI
_DEPon the IVSC device ?
Yes,
Quoted 3 lines
If yes, then yes we can add the IVSC device to
acpi_honor_dep_idand makemei_aceprobe callacpi_dev_clear_dependencies().
But I prefer the powerdown gpio model, because we have to follow the commands sequences as below which is required by firmware, runtime pm is hard to achieve this.
+ /* switch camera sensor ownership to host */
+ ret = ace_set_camera_owner(ACE_CAMERA_HOST);
+ if (ret)
+ goto error;
+
+ /* switch CSI-2 link to host */
+ ret = csi_set_link_owner(CSI_LINK_HOST, callback, context);
+ if (ret)
+ goto release_camera;
+
+ /* configure CSI-2 link */
+ ret = csi_set_link_cfg(nr_of_lanes, link_freq);
+ if (ret)
+ goto release_csi;
BR,
Wentong
Quoted 327 lines
Regards,
Hans
Quoted 72 lines
Quoted 72 lines
Sakari, what do you think of instead of using runtime-pm + devlinks having the IVSC code export a powerdown GPIO to the sensor ?
This also decouples the ivsc powerstate from the sensor power-state which might be useful if we ever want to use some of the more advanced ivsc features, where AFAICT the ivsc fully controls the sensor.
Regards,
Hans
Quoted 58 lines
Quoted 58 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 9:30 PM
To: Wu, Wentong <[email protected]>
Subject: Re: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)Hi Wentong,
Attached are the requested dmesg + acpidump for the Dell Latitude 9420.
Regards,
Hans
Quoted 39 lines · Wentong
Hi
Quoted 37 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 5:28 PMHi,
Quoted 30 lines · Wentong
Quoted 30 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Tuesday, March 7, 2023 5:10 PMHi,
Quoted 21 lines · Wentong
Quoted 21 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Tuesday, March 7, 2023 4:30 PMHi Wentong,
Quoted 12 lines · Wentong
Quoted 12 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Wednesday, March 1, 2023 6:42 PMHi,
On 3/1/23 11:34, Sakari Ailus wrote:
Hi Wentong,On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu
Quoted 5 lines · wrote:
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Intel Visual Sensing Controller (IVSC), codenamed "Clover
Falls", is a companion chip designed to provide secure
and low power vision capability to IA platforms. IVSC is
available in existing commercial platforms from multiple OEMs.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
The primary use case of IVSC is to bring in context awareness.
IVSC interfaces directly with the platform main camera
sensor via a CSI-2 link and processes the image data with
the embedded AI engine. The detected events are sent over
I2C to ISH (Intel Sensor Hub) for additional data fusion
from multiplesensors.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
The fusion results are used to implement advanced use cases
like:
Quoted 169 lines
Quoted 169 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
- Face detection to unlock screen
- Detect user presence to manage backlight setting or
waking up systemSince the Image Processing Unit(IPU) used on the host
processor needs to configure the CSI-2 link in normal
camera usages, the
CSI-2 link and camera sensor can only be used in
mutually-exclusive ways by host IPU and IVSC. By default
the IVSC owns the CSI-2 link and camera sensor. The IPU
driver can take ownership of the CSI-2 link and camera
sensor using interfaces providedQuoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
by this IVSC driver.
Quoted 4 lines
Quoted 4 lines
Switching ownership requires an interface with two
different hardware modules inside IVSC. The software
interface to these modules is via Intel MEI (The IntelManagement Engine) commands.
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 11 lines
Quoted 11 lines
These two hardware modules have two different MEI UUIDs
to enumerate. These hardware
modules are:
- ACE (Algorithm Context Engine): This module is for
algorithm computing when IVSC owns camera sensor. Also
ACE module controls camera sensor's ownership. This
hardware module is used to set ownership
of camera sensor.
- CSI (Camera Serial Interface): This module is used to
route camera sensor data either to IVSC or to host for
IPU driver andapplication.
Quoted 6 lines
Quoted 6 lines
IVSC also provides a privacy mode. When privacy mode is
turned on, camera sensor can't be used. This means that
both ACE and host IPU can't get image data. And when this
mode is turned on, host IPU driver is informed via a
registered callback, so that user can benotified.
Quoted 10 lines
Quoted 10 lines
In summary, to acquire ownership of camera by IPU driver,
first ACE module needs to be informed of ownership and
then to setup MIPI CSI-2 link for the camera sensor and IPU.I thought this for a while and did some research, and I
can suggest the
following:- The IVSC sub-device implements a control for privacy
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
(
V4L2_CID_PRIVACYQuoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
is a good fit).
- Camera sensor access needs to be requested from IVSC
before accessingthe
Quoted 2 lines
Quoted 2 lines
sensor via I²C. The IVSC ownership control needs to be
in theright
Quoted 50 lines
Quoted 50 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
setting for this to work, and device links can be used
for thatpurpose
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
(see device_link_add()). With DL_FLAG_PM_RUNTIME and
DL_FLAG_RPM_ACTIVE,
the supplier devices will be PM runtime resumed before
theconsumer
Quoted 42 lines
Quoted 42 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(camera sensor). As these devices are purely virtual on
host side andhas
Quoted 39 lines
Quoted 39 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
no power state as such, you can use runtime PM callbacks
to transferthe
Quoted 20 lines
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
ownership.
Interesting proposal to use device-links + runtime-pm for
this instead of modelling this as an i2c-mux. FWIW I'm fine
with going this route instead of using an i2c-mux approach.I have been thinking about the i2c-mux approach a bit and
the problem is that we are not really muxing but want to
turn on/off control and AFAIK the i2c-mux framework simply
leaves the mux muxed to the last used i2c-chain, so control
will never be released when the i2ctransfers are done.
Quoted 6 lines
Quoted 6 lines
And if were to somehow modify things (or maybe there
already is some release
callback) then the downside becomes that the i2c-mux core
code operates at the i2c transfer level. So each i2c
read/write would then enable +disavle control.
Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Modelling this using something like runtime pm as such is a
much better fit because then we request control once on
probe / stream-on and release it once we are fully done,
rather then requesting + releasing control once per i2c- transfer.Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe, probably we have to switch to i2c-mux modeling
way.
Quoted 6 lines
Quoted 3 lines
What do you mean? The supplier devices are resumed before the driver's probe is called.
But we setup the link with
device_link_addduring IVSC driver's probe, we can't guarantee driver probe's sequence.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Then maybe we need to do the
device_link_addsomewhere else.sensor's parent is the LJCA I2C device whose driver is being upstream https://www.spinics.net/lists/kernel/msg4702552.htmland
and sensor's power is controlled by IVSC instead of INT3472 if IVSCenabled.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
I believe that the INT3472 code is still involved at least on a Dell Latitude 9420 the INT3472 code still needs to set the clock-enable and the privacy-LED GPIOs otherwise the main camera won't
work.
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 3 lines
So I'm not sure what you mean with "sensor's power is controlled by IVSC instead of INT3472" ?
Could you please share your kernel log and DSDT? Thanks
BR,
WentongQuoted 6 lines
Quoted 6 lines
struct
device_link*device_link_add(struct device *consumer,
struct device *supplier, u32
flags)Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
So probably we have to add above
device_link_addin LJCA I2C's driver, and we can find the consumer(camera sensor) with ACPI API, but the supplier,mei_ace, is mei client device under mei framework and it's dynamically allocated device instead of ACPI device, probably I can find its parent with some ACPI lookup from this LJCA I2C device, but unfortunately mei framework doesn't export the API to find mei client device with its parent bus device(struct
mei_device).Quoted 20 lines
Quoted 20 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
I'm not sure if modeling this
mei_aceas LJCA I2C's runtime power control is acceptable, if yes, probably thismei_acedriver have to go with LJCA I2C device driver.Looking at the ACPI table the sensor ACPI device has 2 _DEP-s listed the I2C controller and the INT3472 device. Since we are already doing similar setup in the INT3472 device that seems like a good place to add the device_link()-s (it can return -EPROBE_DEFER to wait for the
mei_aceto show up).
Quoted 4 lines
Quoted 4 lines
But when the INT3472 code runs, the consumer device does not exist yet and AFAICT the same is true when the LCJA i2c-controller driver is getting
registered.
Quoted 4 lines
Quoted 4 lines
The consumer only exists when the
i2c_clientis instantiated and at that point the sensor driversprobe()method can run immediately and we are too late to add thedevice_link.Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
As a hobby project I have been working on atomisp2 support and I have a similar issue there. There is no INT3472 device there, but there is a
_DSMmethod which needs to be used to figure out which ACPI GPIO resource is reset / powerdown and if the GPIOs are active-lowor active high.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
I have written a little helper function to call the
_DSMand to then turn this into lookups and calldevm_acpi_dev_add_driver_gpios().Since on atomisp2 we cannot use the INT3472 driver to delay the sensor-driver probe and have the INT3472 driver setup the GPIO lookup, at least for the sensor drivers used with
atomisp2 there is going to be a need to add a single line toprobe()likethis:
Quoted 40 lines
Quoted 40 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
v4l2_get_acpi_sensor_info(&i2c_client->dev, NULL);To me it sounds like we need to do something similar here and extend the helper function which I have written (but not yet submitted
upstream) :
Quoted 33 lines
Quoted 33 lines
Quoted 33 lines
Quoted 33 lines
https://github.com/jwrdegoede/linux-
sunxi/commit/e2287979db43d46fa7d354c1bde92eb6219b613dTo also setup the device-links needed for the runtime-pm solution to getting the i2c passed through to the sensor.
Ideally
v4l2_get_acpi_sensor_info()should return void (easier to use in the sensor drivers) but I think it should return an int, so that it can e.g. return -EPROBE_DEFERto wait for themei_ace.Regards,
Hans
Quoted 15 lines
Quoted 15 lines
The mainline kernel delays probing of camera sensors on Intel platforms until the INT3472 driver has probed the INT3472 device on which the sensors have an ACPI
_DEP.This is already used to make sure that clock lookups and regulator info is in place before the sensor's
probe()function runs.So that when the driver does
clk_get()it succeeds and so thatregulator_get()does not end up returning a dummy regulator.So I think the code adding the device_link-s for the IVSC should be added
to:drivers/platform/x86/intel/int3472/discrete.cand then the runtime-resume will happen before the sensor'sprobe()function runs.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Likewise
drivers/platform/x86/intel/int3472/discrete.cshould also ensure that the ivsc driver'sprobe()has run before it calls
acpi_dev_clear_dependencies().Quoted 4 lines
Quoted 4 lines
The
acpi_dev_clear_dependencies()call indiscrete.ctells the ACPI subsystem to go ahead and create the i2c-clients for the sensors and allow the sensor drivers to get loaded and probe thesensor.
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Regards,
Hans
Sakari Ailus
maintainerHi Wentong,
Quoted 70 lines · Wentong
Quoted 64 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 16, 2023 5:04 PMHi,
Quoted 30 lines · Wentong
Quoted 20 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 11:24 PM<re-added the previous Cc list, which I dropped because of the large attachments>
Hi Wentong,
Quoted 6 lines · Wentong
Hi Hans,
Thanks
And AFAICT, there is no IVSC device on your Dell Latitude 9420 where the
platform is based on TGL instead of ADL, and I have never heard IVSC runs on TGL, if no IVSC, INT3472 will control sensor's power.
Quoted 7 lines
Quoted 1 line
And I will double confirm with people who know dell product well tomorrow.
Ah, I was under the impression that there was an IVSC there because:
1. The sensor driver for the used sensor (tries to) poke the IVSC 2. Things did not work without building the IVSC drivers, but that might be due to a dependency on the LCJA GPIO expander instead
Below is your dmesg log, the required SPI controller for IVSC isn't here.
Quoted 8 lines
[ 35.538114] ljca 2-6:1.0: acked sem wait timed out ret:0 timeout:20 ack:0 [ 35.538129] ljca 2-6:1.0: MNG_ENUM_SPI failed ret:-110 len:7 num:0 [ 35.538621] ljca 2-6:1.0: LJCA USB device init success [ 35.538776] usbcore: registered new interface driver ljcaAlso I checked your SSDT, there is no IVSC device and the sensor device depends on
INT3472 instead of IVSC device as on my setup.Ack.
Quoted 1 line
Quoted 1 line
But you might very well be right, that would also explain the "intel vsc not
ready"
Quoted 5 lines
Quoted 5 lines
messages in dmesg.
If with the IVSC case the IVSC controls the power to the sensor too, then another option might be to model the I2C-switch + the power-control as a powerdown GPIO for the sensor, which most sensor
drivers already try to use.
Quoted 6 lines
Quoted 5 lines
The advantage of doing this would be that GPIO lookups can reference the GPIO provider + consumer by device-name so then we don't need to have both devices instantiated at the time of
adding the GPIO lookup. And in that case we could e.g. add the lookup
before registering the I2C controller.Quoted 5 lines
Quoted 2 lines
Can we add IVSC device to
acpi_honor_dep_ids, so that when everything is done duringmei_aceprobe,acpi_dev_clear_dependenciescan make sensorstart probe?
Does the sensor ACPI device node have an ACPI
_DEPon the IVSC device ?
Quoted 8 lines
Yes,
Quoted 3 lines
If yes, then yes we can add the IVSC device to
acpi_honor_dep_idand makemei_aceprobe callacpi_dev_clear_dependencies().But I prefer the powerdown gpio model, because we have to follow the commands sequences as below which is required by firmware, runtime pm is hard to achieve this.
How so?
I don't insist on the runtime PM based solution but I'd rather not have changes to virtually all sensor drivers --- this is an external chip to them.
Quoted 14 lines
(quoted diff) +14
-0+ /* switch camera sensor ownership to host */ + ret = ace_set_camera_owner(ACE_CAMERA_HOST); + if (ret) + goto error; + + /* switch CSI-2 link to host */ + ret = csi_set_link_owner(CSI_LINK_HOST, callback, context); + if (ret) + goto release_camera; + + /* configure CSI-2 link */ + ret = csi_set_link_cfg(nr_of_lanes, link_freq); + if (ret) + goto release_csi;
Signature · 3 lines
Kind regards, Sakari Ailus
Quoted 88 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Friday, March 17, 2023 4:59 PMHi Wentong,
Quoted 24 lines · Wentong
Quoted 24 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 16, 2023 5:04 PMHi,
Quoted 15 lines · Wentong
Quoted 15 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 11:24 PM<re-added the previous Cc list, which I dropped because of the large
attachments>Hi Wentong,
Quoted 2 lines · Wentong
Hi Hans,
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Thanks
And AFAICT, there is no IVSC device on your Dell Latitude 9420 where the
platform is based on TGL instead of ADL, and I have never heard IVSC runs on TGL, if no IVSC, INT3472 will control sensor's power.
Quoted 1 line
And I will double confirm with people who know dell product well
tomorrow.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 6 lines
Ah, I was under the impression that there was an IVSC there because:
1. The sensor driver for the used sensor (tries to) poke the IVSC 2. Things did not work without building the IVSC drivers, but that might be due to a dependency on the LCJA GPIO expander instead
Below is your dmesg log, the required SPI controller for IVSC isn't here.
[ 35.538114] ljca 2-6:1.0: acked sem wait timed out ret:0 timeout:20ack:0
Quoted 36 lines
Quoted 26 lines
Quoted 7 lines
[ 35.538129] ljca 2-6:1.0: MNG_ENUM_SPI failed ret:-110 len:7 num:0 [ 35.538621] ljca 2-6:1.0: LJCA USB device init success [ 35.538776] usbcore: registered new interface driver ljcaAlso I checked your SSDT, there is no IVSC device and the sensor device depends on
INT3472 instead of IVSC device as on my setup.Ack.
Quoted 2 lines
Quoted 2 lines
But you might very well be right, that would also explain the "intel vsc not
ready"
Quoted 6 lines
Quoted 6 lines
messages in dmesg.
If with the IVSC case the IVSC controls the power to the sensor too, then another option might be to model the I2C-switch + the power-control as a powerdown GPIO for the sensor, which most sensor
drivers already try to use.
Quoted 6 lines
Quoted 5 lines
The advantage of doing this would be that GPIO lookups can reference the GPIO provider + consumer by device-name so then we don't need to have both devices instantiated at the time of
adding the GPIO lookup. And in that case we could e.g. add the lookup
before registering the I2C controller.Quoted 6 lines
Quoted 3 lines
Can we add IVSC device to
acpi_honor_dep_ids, so that when everything is done duringmei_aceprobe,acpi_dev_clear_dependenciescan make sensorstart probe?
Does the sensor ACPI device node have an ACPI
_DEPon the IVSC device ?Yes,
Quoted 1 line
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
If yes, then yes we can add the IVSC device to
acpi_honor_dep_idand makemei_aceprobe callacpi_dev_clear_dependencies().But I prefer the powerdown gpio model, because we have to follow the commands sequences as below which is required by firmware, runtime pm is
hard to achieve this.
How so?
But we have to find a way to download commands to firmware following below sequence before writing camera sensor's registers to start camera sensor's steam.
BR,
Wentong
Quoted 23 lines
I don't insist on the runtime PM based solution but I'd rather not have changes to virtually all sensor drivers --- this is an external chip to them.
Quoted 4 lines
(quoted diff) +4
-0+ /* switch camera sensor ownership to host */ + ret = ace_set_camera_owner(ACE_CAMERA_HOST); + if (ret) + goto error;Review
Quoted 10 lines
(quoted diff) +10
-0+ + /* switch CSI-2 link to host */ + ret = csi_set_link_owner(CSI_LINK_HOST, callback, context); + if (ret) + goto release_camera; + + /* configure CSI-2 link */ + ret = csi_set_link_cfg(nr_of_lanes, link_freq); + if (ret) + goto release_csi;--
Kind regards,Sakari Ailus
Quoted 30 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 11:24 PM<re-added the previous Cc list, which I dropped because of the large attachments>
Hi Wentong,
Quoted 5 lines · Wentong
Hi Hans,
Thanks
And AFAICT, there is no IVSC device on your Dell Latitude 9420 where the
platform is based on TGL instead of ADL, and I have never heard IVSC runs on TGL, if no IVSC, INT3472 will control sensor's power.
Quoted 1 line
And I will double confirm with people who know dell product well tomorrow.
Ah, I was under the impression that there was an IVSC there because:
1. The sensor driver for the used sensor (tries to) poke the IVSC 2. Things did not work without building the IVSC drivers, but that might
be due to a dependency on the LCJA GPIO expander insteadBut you might very well be right, that would also explain the "intel vsc not ready" messages in dmesg.
Quoted 8 lines
If with the IVSC case the IVSC controls the power to the sensor too, then another option might be to model the I2C-switch + the power-control as a powerdown GPIO for the sensor, which most sensor drivers already try to use. The advantage of doing this would be that GPIO lookups can reference the GPIO provider + consumer by device-name so then we don't need to have both devices instantiated at the time of
adding the GPIO lookup. And in that case we could e.g. add the lookup
before registering the I2C controller.
Thanks,
So the drivers of sensors connected to IVSC have to add power up/down code.
BR,
Wentong
Quoted 309 lines
Sakari, what do you think of instead of using runtime-pm + devlinks having the IVSC code export a powerdown GPIO to the sensor ?
This also decouples the ivsc powerstate from the sensor power-state which might be useful if we ever want to use some of the more advanced ivsc features, where AFAICT the ivsc fully controls the sensor.
Regards,
Hans
Quoted 83 lines
Quoted 83 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 9:30 PM
To: Wu, Wentong <[email protected]>
Subject: Re: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)Hi Wentong,
Attached are the requested dmesg + acpidump for the Dell Latitude 9420.
Regards,
Hans
Quoted 50 lines · Wentong
Hi
Quoted 48 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 5:28 PMHi,
Quoted 41 lines · Wentong
Quoted 41 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Tuesday, March 7, 2023 5:10 PMHi,
Quoted 32 lines · Wentong
Quoted 32 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Tuesday, March 7, 2023 4:30 PMHi Wentong,
Quoted 23 lines · Wentong
Quoted 23 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Wednesday, March 1, 2023 6:42 PMHi,
Quoted 14 lines · Sakari Ailus
Hi Wentong,
Quoted 11 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover
Falls", is a companion chip designed to provide secure and
low power vision capability to IA platforms. IVSC is
available in existing commercial platforms from multiple OEMs.The primary use case of IVSC is to bring in context awareness.
IVSC interfaces directly with the platform main camera
sensor via a CSI-2 link and processes the image data with
the embedded AI engine. The detected events are sent over
I2C to ISH (Intel Sensor Hub) for additional data fusion
from multiplesensors.
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or
waking up systemSince the Image Processing Unit(IPU) used on the host
processor needs to configure the CSI-2 link in normal
camera usages, the
CSI-2 link and camera sensor can only be used in
mutually-exclusive ways by host IPU and IVSC. By default
the IVSC owns the CSI-2 link and camera sensor. The IPU
driver can take ownership of the CSI-2 link and camera
sensor using interfaces providedQuoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
by this IVSC driver.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Switching ownership requires an interface with two
different hardware modules inside IVSC. The software
interface to these modules is via Intel MEI (The IntelManagement Engine) commands.
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines
Quoted 2 lines
These two hardware modules have two different MEI UUIDs to
enumerate. These hardwaremodules are:
Quoted 4 lines
Quoted 4 lines
- ACE (Algorithm Context Engine): This module is for
algorithm computing when IVSC owns camera sensor. Also ACE
module controls camera sensor's ownership. This hardware
module is used to set ownershipof camera sensor.
Quoted 3 lines
Quoted 3 lines
- CSI (Camera Serial Interface): This module is used to
route camera sensor data either to IVSC or to host for IPU
driver andapplication.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
IVSC also provides a privacy mode. When privacy mode is
turned on, camera sensor can't be used. This means that
both ACE and host IPU can't get image data. And when this
mode is turned on, host IPU driver is informed via a
registered callback, so that user can benotified.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 4 lines
In summary, to acquire ownership of camera by IPU driver,
first ACE module needs to be informed of ownership and then
to setup MIPI CSI-2 link for the camera sensor and IPU.I thought this for a while and did some research, and I can suggest the
following:Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
- The IVSC sub-device implements a control for privacy
(
V4L2_CID_PRIVACYQuoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
is a good fit).
- Camera sensor access needs to be requested from IVSC before accessing
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
sensor via I²C. The IVSC ownership control needs to be in the
right
Quoted 46 lines
Quoted 46 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
setting for this to work, and device links can be used for
thatpurpose
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
(see
device_link_add()). WithDL_FLAG_PM_RUNTIMEand
DL_FLAG_RPM_ACTIVE,Quoted 1 line
the supplier devices will be PM runtime resumed before the
consumer
Quoted 39 lines
Quoted 39 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(camera sensor). As these devices are purely virtual on
host side andhas
Quoted 36 lines
Quoted 36 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
no power state as such, you can use runtime PM callbacks
to transferthe
Quoted 20 lines
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
Quoted 1 line
ownership.
Interesting proposal to use device-links + runtime-pm for this instead of modelling this as an i2c-mux. FWIW I'm fine with going this route instead of using an i2c-mux approach.
I have been thinking about the i2c-mux approach a bit and the problem is that we are not really muxing but want to turn on/off control and AFAIK the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain, so control will never be released when the i2c
transfers are done.
Quoted 6 lines
Quoted 6 lines
And if were to somehow modify things (or maybe there already is some release
callback) then the downside becomes that the i2c-mux core code operates at the i2c transfer level. So each i2c read/write would then enable +disavle control.
Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Modelling this using something like runtime pm as such is a much better fit because then we request control once on probe / stream-on and release it once we are fully done, rather then requesting + releasing control once per i2c- transfer.
Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe, probably we have to switch to i2c-mux modeling
way.
Quoted 4 lines
Quoted 3 lines
What do you mean? The supplier devices are resumed before the driver's probe is called.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
But we setup the link with
device_link_addduring IVSC driver's probe, we can't guarantee driver probe's sequence.Then maybe we need to do the
device_link_addsomewhere else.sensor's parent is the LJCA I2C device whose driver is being upstream https://www.spinics.net/lists/kernel/msg4702552.htmland
and sensor's power is controlled by IVSC instead of INT3472 if IVSCenabled.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
I believe that the INT3472 code is still involved at least on a Dell Latitude 9420 the INT3472 code still needs to set the clock-enable and the privacy-LED GPIOs otherwise the main camera won't
work.
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 3 lines
So I'm not sure what you mean with "sensor's power is controlled by IVSC instead of INT3472" ?
Could you please share your kernel log and DSDT? Thanks
BR,
WentongQuoted 6 lines
Quoted 6 lines
struct
device_link*device_link_add(struct device *consumer,
struct device *supplier, u32
flags)Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
So probably we have to add above
device_link_addin LJCA I2C's driver, and we can find the consumer(camera sensor) with ACPI API, but the supplier,mei_ace, is mei client device under mei framework and it's dynamically allocated device instead of ACPI device, probably I can find its parent with some ACPI lookup from this LJCA I2C device, but unfortunately mei framework doesn't export the API to find mei client device with its parent bus device(struct
mei_device).Quoted 16 lines
Quoted 16 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
I'm not sure if modeling this
mei_aceas LJCA I2C's runtime power control is acceptable, if yes, probably thismei_acedriver have to go with LJCA I2C device driver.Looking at the ACPI table the sensor ACPI device has 2 _DEP-s listed the I2C controller and the INT3472 device. Since we are already doing similar setup in the INT3472 device that seems like a good place to add the device_link()-s (it can return -EPROBE_DEFER to wait for the
mei_aceto show up).
Quoted 4 lines
Quoted 4 lines
But when the INT3472 code runs, the consumer device does not exist yet and AFAICT the same is true when the LCJA i2c-controller driver is getting
registered.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
The consumer only exists when the
i2c_clientis instantiated and at that point the sensor driversprobe()method can run immediately and we are too late to add thedevice_link.As a hobby project I have been working on atomisp2 support and I have a similar issue there. There is no INT3472 device there, but there is a
_DSMmethod which needs to be used to figure out which ACPI GPIO resource is reset / powerdown and if the GPIOs are active-lowor active high.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
I have written a little helper function to call the
_DSMand to then turn this into lookups and calldevm_acpi_dev_add_driver_gpios().Since on atomisp2 we cannot use the INT3472 driver to delay the sensor-driver probe and have the INT3472 driver setup the GPIO lookup, at least for the sensor drivers used with
atomisp2 there is going to be a need to add a single line toprobe()like this:Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
v4l2_get_acpi_sensor_info(&i2c_client->dev, NULL);To me it sounds like we need to do something similar here and extend the helper function which I have written (but not yet submitted
upstream) :
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 46 lines
https://github.com/jwrdegoede/linux-
sunxi/commit/e2287979db43d46fa7d354c1bde92eb6219b613dTo also setup the device-links needed for the runtime-pm solution to getting the i2c passed through to the sensor.
Ideally
v4l2_get_acpi_sensor_info()should return void (easier to use in the sensor drivers) but I think it should return an int, so that it can e.g. return -EPROBE_DEFERto wait for themei_ace.Regards,
Hans
Quoted 10 lines
Quoted 10 lines
The mainline kernel delays probing of camera sensors on Intel platforms until the INT3472 driver has probed the INT3472 device on which the sensors have an ACPI
_DEP.This is already used to make sure that clock lookups and regulator info is in place before the sensor's
probe()function runs.So that when the driver does
clk_get()it succeeds and so thatregulator_get()does not end up returning a dummy regulator.Quoted 8 lines
Quoted 8 lines
So I think the code adding the device_link-s for the IVSC should be added
to:drivers/platform/x86/intel/int3472/discrete.cand then the runtime-resume will happen before the sensor'sprobe()function runs.Likewise
drivers/platform/x86/intel/int3472/discrete.cshould also ensure that the ivsc driver'sprobe()has run before it calls
acpi_dev_clear_dependencies().Quoted 9 lines
Quoted 8 lines
The
acpi_dev_clear_dependencies()call indiscrete.ctells the ACPI subsystem to go ahead and create the i2c-clients for the sensors and allow the sensor drivers to get loaded and probe the sensor.Regards,
Hans
Hi,
Quoted 39 lines · Wentong
Quoted 30 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 11:24 PM<re-added the previous Cc list, which I dropped because of the large attachments>
Hi Wentong,
Quoted 5 lines · Wentong
Hi Hans,
Thanks
And AFAICT, there is no IVSC device on your Dell Latitude 9420 where the
platform is based on TGL instead of ADL, and I have never heard IVSC runs on TGL, if no IVSC, INT3472 will control sensor's power.
Quoted 1 line
And I will double confirm with people who know dell product well tomorrow.
Ah, I was under the impression that there was an IVSC there because:
1. The sensor driver for the used sensor (tries to) poke the IVSC 2. Things did not work without building the IVSC drivers, but that might
be due to a dependency on the LCJA GPIO expander insteadBut you might very well be right, that would also explain the "intel vsc not ready" messages in dmesg.
Quoted 8 lines
If with the IVSC case the IVSC controls the power to the sensor too, then another option might be to model the I2C-switch + the power-control as a powerdown GPIO for the sensor, which most sensor drivers already try to use. The advantage of doing this would be that GPIO lookups can reference the GPIO provider + consumer by device-name so then we don't need to have both devices instantiated at the time of
adding the GPIO lookup. And in that case we could e.g. add the lookup
before registering the I2C controller.
Quoted 3 lines
Thanks,
So the drivers of sensors connected to IVSC have to add power up/down code.
If we go the model it as a powerdown GPIO route, yes. Note most sensor drivers already have support for this since this is used in the INT3472 case already.
Regards,
Hans
Quoted 313 lines
BR,
WentongQuoted 309 lines
Sakari, what do you think of instead of using runtime-pm + devlinks having the IVSC code export a powerdown GPIO to the sensor ?
This also decouples the ivsc powerstate from the sensor power-state which might be useful if we ever want to use some of the more advanced ivsc features, where AFAICT the ivsc fully controls the sensor.
Regards,
Hans
Quoted 83 lines
Quoted 83 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 9:30 PM
To: Wu, Wentong <[email protected]>
Subject: Re: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)Hi Wentong,
Attached are the requested dmesg + acpidump for the Dell Latitude 9420.
Regards,
Hans
Quoted 50 lines · Wentong
Hi
Quoted 48 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Thursday, March 9, 2023 5:28 PMHi,
Quoted 41 lines · Wentong
Quoted 41 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Tuesday, March 7, 2023 5:10 PMHi,
Quoted 32 lines · Wentong
Quoted 32 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Tuesday, March 7, 2023 4:30 PMHi Wentong,
Quoted 23 lines · Wentong
Quoted 23 lines
-----Original Message-----
From: Hans de Goede <[email protected]>
Sent: Wednesday, March 1, 2023 6:42 PMHi,
Quoted 14 lines · Sakari Ailus
Hi Wentong,
On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
Intel Visual Sensing Controller (IVSC), codenamed "Clover
Falls", is a companion chip designed to provide secure and
low power vision capability to IA platforms. IVSC is
available in existing commercial platforms from multiple OEMs.The primary use case of IVSC is to bring in context awareness.
IVSC interfaces directly with the platform main camera
sensor via a CSI-2 link and processes the image data with
the embedded AI engine. The detected events are sent over
I2C to ISH (Intel Sensor Hub) for additional data fusion
from multiplesensors.
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or
waking up systemSince the Image Processing Unit(IPU) used on the host
processor needs to configure the CSI-2 link in normal
camera usages, the
CSI-2 link and camera sensor can only be used in
mutually-exclusive ways by host IPU and IVSC. By default
the IVSC owns the CSI-2 link and camera sensor. The IPU
driver can take ownership of the CSI-2 link and camera
sensor using interfaces providedQuoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
by this IVSC driver.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Switching ownership requires an interface with two
different hardware modules inside IVSC. The software
interface to these modules is via Intel MEI (The IntelManagement Engine) commands.
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines
These two hardware modules have two different MEI UUIDs to
enumerate. These hardwaremodules are:
Quoted 4 lines
- ACE (Algorithm Context Engine): This module is for
algorithm computing when IVSC owns camera sensor. Also ACE
module controls camera sensor's ownership. This hardware
module is used to set ownershipof camera sensor.
Quoted 3 lines
- CSI (Camera Serial Interface): This module is used to
route camera sensor data either to IVSC or to host for IPU
driver andapplication.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
IVSC also provides a privacy mode. When privacy mode is
turned on, camera sensor can't be used. This means that
both ACE and host IPU can't get image data. And when this
mode is turned on, host IPU driver is informed via a
registered callback, so that user can benotified.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
In summary, to acquire ownership of camera by IPU driver,
first ACE module needs to be informed of ownership and then
to setup MIPI CSI-2 link for the camera sensor and IPU.I thought this for a while and did some research, and I can
suggest the
following:Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
- The IVSC sub-device implements a control for privacy
(
V4L2_CID_PRIVACYQuoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
is a good fit).
- Camera sensor access needs to be requested from IVSC
before accessingthe
Quoted 1 line
Quoted 1 line
Quoted 1 line
sensor via I²C. The IVSC ownership control needs to be in the
right
Quoted 46 lines
Quoted 46 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
setting for this to work, and device links can be used for
thatpurpose
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
(see device_link_add()). With DL_FLAG_PM_RUNTIME and
DL_FLAG_RPM_ACTIVE,Quoted 1 line
the supplier devices will be PM runtime resumed before the
consumer
Quoted 39 lines
Quoted 39 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(camera sensor). As these devices are purely virtual on
host side andhas
Quoted 36 lines
Quoted 36 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
no power state as such, you can use runtime PM callbacks
to transferthe
Quoted 20 lines
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
Quoted 1 line
ownership.
Interesting proposal to use device-links + runtime-pm for this instead of modelling this as an i2c-mux. FWIW I'm fine with going this route instead of using an i2c-mux approach.
I have been thinking about the i2c-mux approach a bit and the problem is that we are not really muxing but want to turn on/off control and AFAIK the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain, so control will never be released when the i2c
transfers are done.
Quoted 6 lines
Quoted 6 lines
And if were to somehow modify things (or maybe there already is some release
callback) then the downside becomes that the i2c-mux core code operates at the i2c transfer level. So each i2c read/write would then enable +disavle control.
Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Modelling this using something like runtime pm as such is a much better fit because then we request control once on probe / stream-on and release it once we are fully done, rather then requesting + releasing control once per i2c- transfer.
Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe, probably we have to switch to i2c-mux modeling
way.
Quoted 4 lines
Quoted 3 lines
What do you mean? The supplier devices are resumed before the driver's probe is called.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
But we setup the link with
device_link_addduring IVSC driver's probe, we can't guarantee driver probe's sequence.Then maybe we need to do the
device_link_addsomewhere else.sensor's parent is the LJCA I2C device whose driver is being upstream https://www.spinics.net/lists/kernel/msg4702552.htmland
and sensor's power is controlled by IVSC instead of INT3472 if IVSCenabled.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
I believe that the INT3472 code is still involved at least on a Dell Latitude 9420 the INT3472 code still needs to set the clock-enable and the privacy-LED GPIOs otherwise the main camera won't
work.
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 3 lines
So I'm not sure what you mean with "sensor's power is controlled by IVSC instead of INT3472" ?
Could you please share your kernel log and DSDT? Thanks
BR,
WentongQuoted 6 lines
Quoted 6 lines
struct
device_link*device_link_add(struct device *consumer,
struct device *supplier, u32
flags)Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
So probably we have to add above
device_link_addin LJCA I2C's driver, and we can find the consumer(camera sensor) with ACPI API, but the supplier,mei_ace, is mei client device under mei framework and it's dynamically allocated device instead of ACPI device, probably I can find its parent with some ACPI lookup from this LJCA I2C device, but unfortunately mei framework doesn't export the API to find mei client device with its parent bus device(struct
mei_device).Quoted 16 lines
Quoted 16 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
I'm not sure if modeling this
mei_aceas LJCA I2C's runtime power control is acceptable, if yes, probably thismei_acedriver have to go with LJCA I2C device driver.Looking at the ACPI table the sensor ACPI device has 2 _DEP-s listed the I2C controller and the INT3472 device. Since we are already doing similar setup in the INT3472 device that seems like a good place to add the device_link()-s (it can return -EPROBE_DEFER to wait for the
mei_aceto show up).
Quoted 4 lines
Quoted 4 lines
But when the INT3472 code runs, the consumer device does not exist yet and AFAICT the same is true when the LCJA i2c-controller driver is getting
registered.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
The consumer only exists when the
i2c_clientis instantiated and at that point the sensor driversprobe()method can run immediately and we are too late to add thedevice_link.As a hobby project I have been working on atomisp2 support and I have a similar issue there. There is no INT3472 device there, but there is a
_DSMmethod which needs to be used to figure out which ACPI GPIO resource is reset / powerdown and if the GPIOs are active-lowor active high.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
I have written a little helper function to call the
_DSMand to then turn this into lookups and calldevm_acpi_dev_add_driver_gpios().Since on atomisp2 we cannot use the INT3472 driver to delay the sensor-driver probe and have the INT3472 driver setup the GPIO lookup, at least for the sensor drivers used with
atomisp2 there is going to be a need to add a single line toprobe()like this:Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
v4l2_get_acpi_sensor_info(&i2c_client->dev, NULL);To me it sounds like we need to do something similar here and extend the helper function which I have written (but not yet submitted
upstream) :
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 46 lines
https://github.com/jwrdegoede/linux-
sunxi/commit/e2287979db43d46fa7d354c1bde92eb6219b613dTo also setup the device-links needed for the runtime-pm solution to getting the i2c passed through to the sensor.
Ideally
v4l2_get_acpi_sensor_info()should return void (easier to use in the sensor drivers) but I think it should return an int, so that it can e.g. return -EPROBE_DEFERto wait for themei_ace.Regards,
Hans
Quoted 10 lines
Quoted 10 lines
The mainline kernel delays probing of camera sensors on Intel platforms until the INT3472 driver has probed the INT3472 device on which the sensors have an ACPI
_DEP.This is already used to make sure that clock lookups and regulator info is in place before the sensor's
probe()function runs.So that when the driver does
clk_get()it succeeds and so thatregulator_get()does not end up returning a dummy regulator.Quoted 8 lines
Quoted 8 lines
So I think the code adding the device_link-s for the IVSC should be added
to:drivers/platform/x86/intel/int3472/discrete.cand then the runtime-resume will happen before the sensor'sprobe()function runs.Likewise
drivers/platform/x86/intel/int3472/discrete.cshould also ensure that the ivsc driver'sprobe()has run before it calls
acpi_dev_clear_dependencies().Quoted 9 lines
Quoted 8 lines
The
acpi_dev_clear_dependencies()call indiscrete.ctells the ACPI subsystem to go ahead and create the i2c-clients for the sensors and allow the sensor drivers to get loaded and probe the sensor.Regards,
Hans
Quoted 46 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Wednesday, March 1, 2023 6:34 PMHi Wentong,
Quoted 22 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
systemSince the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces provided by this IVSC driver.
Quoted 4 lines
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware
modules are:
Quoted 3 lines
- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership ofcamera sensor.
Quoted 8 lines
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, host IPU driver is informed via a registered callback, so that user can be notified.
Quoted 6 lines
Quoted 3 lines
In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
I thought this for a while and did some research, and I can suggest the following:
Thanks a lot
Quoted 3 lines
- The IVSC sub-device implements a control for privacy (
V4L2_CID_PRIVACYis a good fit).
Agree, thanks
Quoted 11 lines
- Camera sensor access needs to be requested from IVSC before accessing the sensor via I²C. The IVSC ownership control needs to be in the right setting for this to work, and device links can be used for that purpose (see
device_link_add()). WithDL_FLAG_PM_RUNTIMEandDL_FLAG_RPM_ACTIVE,
the supplier devices will be PM runtime resumed before the consumer (camera sensor). As these devices are purely virtual on host side and has no power state as such, you can use runtime PM callbacks to transfer the ownership.
That's pretty cool, and the device link will be setup during mei_ace's probe with device_link_add, but we should guarantee there is no i2c transfer during sensor driver's probe. And most of upstream sensor drivers don't initial i2c transfer during probe, but I have no idea what will be like for our upstreaming sensor drivers.
Quoted 2 lines
- The CSI-2 configuration should take place when streaming starts on the sensor (as well as IVSC).
IPU driver should make sure start sensor streaming first and then IVSC sub-dev, which will match the command downloading sequence required by firmware.
Quoted 3 lines
- Device links need to be set up via IPU bridge which now is called CIO2 bridge (
cio2-bridge.c).
Reviewing the code
Quoted 7 lines
Any questions, comments?
--
Kind regards,Sakari Ailus