media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)
Subsystems (3)
v3 cover media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up system
Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces exported via v4l2 sub-device.
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware modules are:
- ACE (Algorithm Context Engine): This module is for algorithm computing
when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.
IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, users are informed via v4l2 control API.
In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
Implementation:
There are two different drivers to handle ACE and CSI hardware modules inside IVSC.
- ivsc_csi: MEI client driver to send commands and receive notifications
from CSI module.
- ivsc_ace: MEI client driver to send commands and get status from ACE
module.
Interface is exposed via v4l2 sub-devcie APIs to acquire and release camera sensor and CSI-2 link.
Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
-----------------------------------------------------------------------------
| Host Processor |
| |
| ----------------- ----------------- --------------- |
| | | | | | | I2C |
| | IPU | | ISH | |camera driver|--| |
| | | | | | | | |
| ----------------- ----------------- --------------- | |
| | | | | |
| | | --------------- | |
| | | | | | |
| | | | IVSC driver | | |
| | | | | | |
| | | --------------- | |
| | | | | |
----------------|-----------------------|----------------------|---------|---
| CSI | I2C |SPI |
| | | |
----------------|-----------------------|---------------- | |
| IVSC | | | |
| | | | |
| ----------------- ----------------- | | |
| | | | | | | |
| | CSI | | ACE | |------| |
| | | | | | |
| ----------------- ----------------- | |
| | | I2C | |
----------------|-----------------------|---------------- |
| CSI | |
| | |
-------------------------------- |
| | I2C |
| camera sensor |-----------------------------|
| |
--------------------------------Wentong Wu (3):
media: pci: intel: ivsc: Add CSI submodule
media: pci: intel: ivsc: Add ACE submodule
ACPI: delay enumeration of devices with a _DEP pointing to INTC1059
device
drivers/acpi/scan.c | 1 + drivers/media/pci/Kconfig | 1 + drivers/media/pci/intel/Makefile | 2 + drivers/media/pci/intel/ivsc/Kconfig | 12 + drivers/media/pci/intel/ivsc/Makefile | 10 + drivers/media/pci/intel/ivsc/csi_bridge.c | 332 +++++++++++++ drivers/media/pci/intel/ivsc/csi_bridge.h | 122 +++++ drivers/media/pci/intel/ivsc/mei_ace.c | 534 ++++++++++++++++++++ drivers/media/pci/intel/ivsc/mei_csi.c | 775 ++++++++++++++++++++++++++++++ 9 files changed, 1789 insertions(+) create mode 100644 drivers/media/pci/intel/ivsc/Kconfig create mode 100644 drivers/media/pci/intel/ivsc/Makefile create mode 100644 drivers/media/pci/intel/ivsc/csi_bridge.c create mode 100644 drivers/media/pci/intel/ivsc/csi_bridge.h create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
Signature · 1 line
2.7.4
PATCH v3 1/3 media: pci: intel: ivsc: Add CSI submodule
CSI is a submodule of IVSC which can route camera sensor data to the outbound MIPI CSI-2 interface.
The interface communicating with firmware is via MEI. There is a separate MEI UUID, which this driver uses to enumerate.
To route camera sensor data to host, the information of link frequency and number of data lanes is sent to firmware by sending MEI command when starting stream.
CSI also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both IVSC and host Image Processing Unit(IPU) can't get image data. And when this mode is turned on, host Image Processing Unit(IPU) driver is informed via v4l2 control callback, so that user can be notified.
--- drivers/media/pci/Kconfig | 1 + drivers/media/pci/intel/Makefile | 2 + drivers/media/pci/intel/ivsc/Kconfig | 12 + drivers/media/pci/intel/ivsc/Makefile | 7 + drivers/media/pci/intel/ivsc/csi_bridge.c | 332 +++++++++++++ drivers/media/pci/intel/ivsc/csi_bridge.h | 122 +++++ drivers/media/pci/intel/ivsc/mei_csi.c | 775 ++++++++++++++++++++++++++++++ 7 files changed, 1251 insertions(+) create mode 100644 drivers/media/pci/intel/ivsc/Kconfig create mode 100644 drivers/media/pci/intel/ivsc/Makefile create mode 100644 drivers/media/pci/intel/ivsc/csi_bridge.c create mode 100644 drivers/media/pci/intel/ivsc/csi_bridge.h create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
drivers/media/pci/Kconfig +1 -0
@@ -72,6 +72,7 @@ config VIDEO_PCI_SKELETON when developing new drivers. source "drivers/media/pci/intel/ipu3/Kconfig" +source "drivers/media/pci/intel/ivsc/Kconfig" endif #MEDIA_PCI_SUPPORT endif #PCI
drivers/media/pci/intel/Makefile +2 -0
@@ -4,3 +4,5 @@ # obj-y += ipu3/ + +obj-$(CONFIG_INTEL_VSC) += ivsc/
drivers/media/pci/intel/ivsc/Kconfig new file +12 -0
@@ -0,0 +1,12 @@ +# SPDX-License-Identifier: GPL-2.0-only +# Copyright (C) 2023, Intel Corporation. All rights reserved. + +config INTEL_VSC + tristate "Intel Visual Sensing Controller" + depends on INTEL_MEI + help + This adds support for Intel Visual Sensing Controller (IVSC). + + Enables the IVSC firmware services required for controlling + camera sensor ownership and CSI-2 link through Image Processing + Unit(IPU) driver of Intel.
drivers/media/pci/intel/ivsc/Makefile new file +7 -0
@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +# Copyright (C) 2023, Intel Corporation. All rights reserved. + +obj-$(CONFIG_INTEL_VSC) += ivsc-csi.o +ivsc-csi-y += mei_csi.o +ivsc-csi-y += csi_bridge.o
drivers/media/pci/intel/ivsc/csi_bridge.c new file +332 -0
332 lines not on this page · raw message
drivers/media/pci/intel/ivsc/csi_bridge.h new file +122 -0
122 lines not on this page · raw message
drivers/media/pci/intel/ivsc/mei_csi.c new file +775 -0
775 lines not on this page · raw message
Signature · 1 line
2.7.4
kernel test robot
build botRe: [PATCH v3 1/3] media: pci: intel: ivsc: Add CSI submodule
Thank you for the patch! Perhaps something to improve:
Hi Wentong,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on media-tree/master]
[also build test WARNING on rafael-pm/linux-next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Wentong-Wu/media-pci-intel-ivsc-Add-CSI-submodule/20230327-142435 base: git://linuxtv.org/media_tree.git master patch link: https://lore.kernel.org/r/1679898188-14426-2-git-send-email-wentong.wu%40intel.com patch subject: [PATCH v3 1/3] media: pci: intel: ivsc: Add CSI submodule config: i386-allyesconfig (https://download.01.org/0day-ci/archive/20230327/[email protected]/config) compiler: gcc-11 (Debian 11.3.0-8) 11.3.0
reproduce (this is a W=1 build):
# https://github.com/intel-lab-lkp/linux/commit/413726a06ca6f775b34425ed81e996b9b7237d31
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Wentong-Wu/media-pci-intel-ivsc-Add-CSI-submodule/20230327-142435 git checkout 413726a06ca6f775b34425ed81e996b9b7237d31
# save the config file
mkdir build_dir && cp config build_dir/.config
make W=1 O=build_dir ARCH=i386 olddefconfig
make W=1 O=build_dir ARCH=i386 SHELL=/bin/bash drivers/media/pci/intel/ivsc/
If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <[email protected]>
| Link: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
drivers/media/pci/intel/ivsc/csi_bridge.c: In function 'csi_bridge_unregister_sensor':
drivers/media/pci/intel/ivsc/csi_bridge.c:179:9: error: implicit declaration of function 'software_node_unregister_nodes'; did you mean 'software_node_unregister'? [-Werror=implicit-function-declaration]
179 | software_node_unregister_nodes(sensor->swnodes);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| software_node_unregister
drivers/media/pci/intel/ivsc/csi_bridge.c: In function 'csi_bridge_connect_sensor':
drivers/media/pci/intel/ivsc/csi_bridge.c:231:15: error: implicit declaration of function 'software_node_register_nodes'; did you mean 'software_node_register'? [-Werror=implicit-function-declaration]
231 | ret = software_node_register_nodes(sensor->swnodes);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
| software_node_register
drivers/media/pci/intel/ivsc/csi_bridge.c: At top level:
>> drivers/media/pci/intel/ivsc/csi_bridge.c:284:5: warning: no previous prototype for 'csi_bridge_init' [-Wmissing-prototypes] 284 | int csi_bridge_init(struct mei_cl_device *csi_dev)
| ^~~~~~~~~~~~~~~
cc1: some warnings being treated as errors
vim +/csi_bridge_init +284 drivers/media/pci/intel/ivsc/csi_bridge.c
283
> 284 int csi_bridge_init(struct mei_cl_device *csi_dev)
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests
kernel test robot
build botRe: [PATCH v3 1/3] media: pci: intel: ivsc: Add CSI submodule
In reply to Wentong WuThank you for the patch! Perhaps something to improve:
Hi Wentong,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on media-tree/master]
[also build test WARNING on rafael-pm/linux-next sailus-media-tree/streams linus/master v6.3-rc4 next-20230328] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Wentong-Wu/media-pci-intel-ivsc-Add-CSI-submodule/20230327-142435 base: git://linuxtv.org/media_tree.git master patch link: https://lore.kernel.org/r/1679898188-14426-2-git-send-email-wentong.wu%40intel.com patch subject: [PATCH v3 1/3] media: pci: intel: ivsc: Add CSI submodule config: i386-allyesconfig (https://download.01.org/0day-ci/archive/20230328/[email protected]/config) compiler: clang version 14.0.6 (https://github.com/llvm/llvm-project f28c006a5895fc0e329fe15fead81e37457cb1d1)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/413726a06ca6f775b34425ed81e996b9b7237d31
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Wentong-Wu/media-pci-intel-ivsc-Add-CSI-submodule/20230327-142435
git checkout 413726a06ca6f775b34425ed81e996b9b7237d31
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=i386 olddefconfig
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=i386 SHELL=/bin/bash drivers/media/i2c/ drivers/media/pci/intel/ivsc/ drivers/net/usb/ net/netfilter/
If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <[email protected]>
| Link: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
drivers/media/pci/intel/ivsc/csi_bridge.c:179:2: error: implicit declaration of function 'software_node_unregister_nodes' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
software_node_unregister_nodes(sensor->swnodes);
^
drivers/media/pci/intel/ivsc/csi_bridge.c:231:8: error: implicit declaration of function 'software_node_register_nodes' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
ret = software_node_register_nodes(sensor->swnodes);
^
drivers/media/pci/intel/ivsc/csi_bridge.c:249:2: error: implicit declaration of function 'software_node_unregister_nodes' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
software_node_unregister_nodes(sensor->swnodes);
^
>> drivers/media/pci/intel/ivsc/csi_bridge.c:284:5: warning: no previous prototype for function 'csi_bridge_init' [-Wmissing-prototypes]
int csi_bridge_init(struct mei_cl_device *csi_dev)
^ drivers/media/pci/intel/ivsc/csi_bridge.c:284:1: note: declare 'static' if the function is not intended to be used outside of this translation unit int csi_bridge_init(struct mei_cl_device *csi_dev)
^
static
1 warning and 3 errors generated.
vim +/csi_bridge_init +284 drivers/media/pci/intel/ivsc/csi_bridge.c
283
> 284 int csi_bridge_init(struct mei_cl_device *csi_dev)
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests
Sakari Ailus
maintainerRe: [PATCH v3 1/3] media: pci: intel: ivsc: Add CSI submodule
In reply to Wentong WuHi Wentong,
Quoted 18 lines · Wentong Wu
CSI is a submodule of IVSC which can route camera sensor data to the outbound MIPI CSI-2 interface.
The interface communicating with firmware is via MEI. There is a separate MEI UUID, which this driver uses to enumerate.
To route camera sensor data to host, the information of link frequency and number of data lanes is sent to firmware by sending MEI command when starting stream.
CSI also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both IVSC and host Image Processing Unit(IPU) can't get image data. And when this mode is turned on, host Image Processing Unit(IPU) driver is informed via v4l2 control callback, so that user can be notified.
Signed-off-by Wentong Wu <[email protected]>
Quoted 14 lines
--- drivers/media/pci/Kconfig | 1 + drivers/media/pci/intel/Makefile | 2 + drivers/media/pci/intel/ivsc/Kconfig | 12 + drivers/media/pci/intel/ivsc/Makefile | 7 + drivers/media/pci/intel/ivsc/csi_bridge.c | 332 +++++++++++++ drivers/media/pci/intel/ivsc/csi_bridge.h | 122 +++++ drivers/media/pci/intel/ivsc/mei_csi.c | 775 ++++++++++++++++++++++++++++++ 7 files changed, 1251 insertions(+) create mode 100644 drivers/media/pci/intel/ivsc/Kconfig create mode 100644 drivers/media/pci/intel/ivsc/Makefile create mode 100644 drivers/media/pci/intel/ivsc/csi_bridge.c create mode 100644 drivers/media/pci/intel/ivsc/csi_bridge.h create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
I'm commenting just mei_csi this time...
Quoted 198 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +192
-0@@ -0,0 +1,775 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright (C) 2023 Intel Corporation. All rights reserved. +* Intel Visual Sensing Controller CSI Linux driver +*/ + +/* +* To set ownership of CSI-2 link and to configure CSI-2 link, there +* are specific commands, which are sent via MEI protocol. The send +* command function uses "completion" as a synchronization mechanism. +* The response for command is received via a mei callback which wakes +* up the caller. There can be only one outstanding command at a time. +*/ + +#include <linux/completion.h> +#include <linux/delay.h> +#include <linux/kernel.h> +#include <linux/math64.h> +#include <linux/mei_cl_bus.h> +#include <linux/module.h> +#include <linux/mutex.h> +#include <linux/pm_runtime.h> +#include <linux/slab.h> +#include <linux/units.h> +#include <linux/uuid.h> +#include <linux/workqueue.h> + +#include <media/v4l2-async.h> +#include <media/v4l2-ctrls.h> +#include <media/v4l2-fwnode.h> +#include <media/v4l2-subdev.h> + +#define MEI_CSI_DRIVER_NAME "ivsc_csi" + +/* the 5s used here is based on experiment */ +#define CSI_CMD_TIMEOUT (5 * HZ) +/* to setup CSI-2 link an extra delay needed and determined experimentally */ +#define CSI_FW_READY_DELAY_MS 100 +/* link frequency unit is 100kHz */ +#define CSI_LINK_FREQ (x) ((u32)(div_u64(x, 100 * HZ_PER_KHZ))) + +/* +* identify the command id supported by firmware +* IPC, as well as the privacy notification id +* used when processing privacy event. +*/ +enum csi_cmd_id { +/* used to set csi ownership */ +CSI_SET_OWNER =0 , + +/* used to configure CSI-2 link */ +CSI_SET_CONF =2 , + +/* privacy notification id used when privacy state changes */ +CSI_PRIVACY_NOTIF =6 , +}; + +/* CSI-2 link ownership definition */ +enum csi_link_owner { +CSI_LINK_IVSC , +CSI_LINK_HOST , +}; + +/* privacy status definition */ +enum ivsc_privacy_status { +CSI_PRIVACY_OFF , +CSI_PRIVACY_ON , +CSI_PRIVACY_MAX , +}; + +enum csi_pads { +CSI_PAD_SOURCE , +CSI_PAD_SINK , +CSI_NUM_PADS +}; + +/* configuration of the CSI-2 link between host and IVSC */ +struct csi_link_cfg { +/* number of data lanes used on the CSI-2 link */ +u32 nr_of_lanes; + +/* frequency of the CSI-2 link */ +u32 link_freq; + +/* for future use */ +u32 rsvd[2 ]; +} __packed; + +/* CSI command structure */ +struct csi_cmd { +u32 cmd_id; +union _cmd_param { +u32 param; +struct csi_link_cfg conf; + } param; +} __packed; + +/* CSI notification structure */ +struct csi_notif { +u32 cmd_id; +int status; +union _resp_cont { +u32 cont; +struct csi_link_cfg conf; + } cont; +} __packed; + +struct mei_csi { +struct mei_cl_device *cldev; + +/* command response */ +struct csi_notif cmd_response; +/* used to wait for command response from firmware */ +struct completion cmd_completion; +/* protect command download */ +struct mutex lock; + +struct v4l2_subdev subdev; +struct v4l2_subdev *remote; +struct v4l2_async_notifier notifier; +struct v4l2_ctrl_handler ctrl_handler; +struct v4l2_ctrl *privacy_ctrl; +unsigned int remote_pad; +/* start streaming or not */ +int streaming; + +struct media_pad pads[CSI_NUM_PADS ]; +struct v4l2_mbus_framefmt format_mbus[CSI_NUM_PADS ]; + +/* number of data lanes used on the CSI-2 link */ +u32 nr_of_lanes; +/* frequency of the CSI-2 link */ +u64 link_freq; + +/* privacy status */ +enum ivsc_privacy_status status; +}; + +static const struct v4l2_mbus_framefmt mei_csi_format_mbus_default = { + .width =1 , + .height =1 , + .code =MEDIA_BUS_FMT_Y8_1X8 , + .field =V4L2_FIELD_NONE , +}; + +int csi_bridge_init (struct mei_cl_device *csi_dev); + +static inline struct mei_csi *notifier_to_csi (struct v4l2_async_notifier *n) +{ + return container_of(n, struct mei_csi, notifier); +} + +static inline struct mei_csi *sd_to_csi (struct v4l2_subdev *sd) +{ + return container_of(sd, struct mei_csi, subdev); +} + +/* send a command to firmware and mutex must be held by caller */ +static int mei_csi_send (struct mei_csi *csi,u8 *buf,size_t len) +{ +struct csi_cmd *cmd = (struct csi_cmd *)buf; +int ret; + +reinit_completion (&csi->cmd_completion); + + ret =mei_cldev_send (csi->cldev, buf, len); +if (ret <0 ) + goto out; + + ret =wait_for_completion_killable_timeout (&csi->cmd_completion, +CSI_CMD_TIMEOUT ); +if (ret <0 ) { + goto out; + }else if (!ret) { + ret = -ETIMEDOUT ; + goto out; + } + +/* command response status */ + ret = csi->cmd_response.status; +if (ret) { + ret = -EINVAL ; + goto out; + } + +if (csi->cmd_response.cmd_id != cmd->cmd_id) + ret = -EINVAL ; + +out: +return ret; +} +
drivers/media/pci/intel/ivsc/mei_csi.c line 202
Quoted 10 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +10
-0+/* set CSI-2 link ownership */ +static int csi_set_link_owner (struct mei_csi *csi,enum csi_link_owner owner) +{ +struct csi_cmd cmd = {0 }; +size_t cmd_size; +int ret; + + cmd.cmd_id =CSI_SET_OWNER ; + cmd.param.param = owner; + cmd_size =sizeof (cmd.cmd_id) +sizeof (cmd.param.param);
In some cases you're using memset and in others not. If you don't need memset, I'd prefer assigning the fields in variable declaration instead.
Quoted 11 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +11
-0+ +mutex_lock (&csi->lock); + + ret =mei_csi_send (csi, (u8 *)&cmd, cmd_size); + +mutex_unlock (&csi->lock); + +return ret; +} + +/* configure CSI-2 link between host and IVSC */
drivers/media/pci/intel/ivsc/mei_csi.c line 223
Quoted 10 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +10
-0+static int csi_set_link_cfg (struct mei_csi *csi) +{ +struct csi_cmd cmd = {0 }; +size_t cmd_size; +int ret; + + cmd.cmd_id =CSI_SET_CONF ; + cmd.param.conf.nr_of_lanes = csi->nr_of_lanes; + cmd.param.conf.link_freq =CSI_LINK_FREQ (csi->link_freq); + cmd_size =sizeof (cmd.cmd_id) +sizeof (cmd.param.conf);
Ditto.
Quoted 53 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +53
-0+ +mutex_lock (&csi->lock); + + ret =mei_csi_send (csi, (u8 *)&cmd, cmd_size); +/* +* wait configuration ready if download success. placing +* delay under mutex is to make sure current command flow +* completed before starting a possible new one. +*/ +if (!ret) +msleep (CSI_FW_READY_DELAY_MS ); + +mutex_unlock (&csi->lock); + +return ret; +} + +/* callback for receive */ +static void mei_csi_rx (struct mei_cl_device *cldev) +{ +struct mei_csi *csi =mei_cldev_get_drvdata (cldev); +struct csi_notif notif = {0 }; +int ret; + + ret =mei_cldev_recv (cldev, (u8 *)¬if,sizeof (notif)); +if (ret <0 ) { +dev_err (&cldev->dev,"recv error: %d\n" , ret); +return ; + } + +switch (notif.cmd_id) { +case CSI_PRIVACY_NOTIF : +if (notif.cont.cont <CSI_PRIVACY_MAX ) { + csi->status = notif.cont.cont; +v4l2_ctrl_s_ctrl (csi->privacy_ctrl, csi->status); + } +break ; +case CSI_SET_OWNER : +case CSI_SET_CONF : +memcpy (&csi->cmd_response, ¬if, ret); + +complete (&csi->cmd_completion); +break ; +default : +break ; + } +} + +static int mei_csi_pre_streamon (struct v4l2_subdev *sd,u32 flags) +{ +struct v4l2_querymenu qm = { .id =V4L2_CID_LINK_FREQ , }; +struct v4l2_mbus_config mbus_config = {0 }; +struct mei_csi *csi =sd_to_csi (sd);
drivers/media/pci/intel/ivsc/mei_csi.c line 286
Quoted 10 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +10
-0+struct v4l2_ctrl *ctrl; +int ret =0 ; + +if (!csi->remote) +return -ENODEV ; + + ret =v4l2_subdev_call (csi->remote, pad, get_mbus_config, + csi->remote_pad, &mbus_config); +if (ret) +return ret;
You're already parsing the endpoint in probe(). Do you need this, assuming the sensor has a static lane configuration? Virtually all do.
Could this all be done via the s_stream() op instead?
Quoted 36 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +36
-0+ +if (mbus_config.type !=V4L2_MBUS_CSI2_DPHY ) +return -EINVAL ; + + csi->nr_of_lanes = mbus_config.bus.mipi_csi2.num_data_lanes; + + ctrl =v4l2_ctrl_find (csi->remote->ctrl_handler,V4L2_CID_LINK_FREQ ); +if (!ctrl) +return -EINVAL ; + qm.index =v4l2_ctrl_g_ctrl (ctrl); + + ret =v4l2_querymenu (csi->remote->ctrl_handler, &qm); +if (ret) +return ret; + + csi->link_freq = qm.value; + +return ret; +} + +static int mei_csi_set_stream (struct v4l2_subdev *sd,int enable) +{ +struct mei_csi *csi =sd_to_csi (sd); +int ret =0 ; + +if (enable && csi->streaming ==0 ) { +/* switch CSI-2 link to host */ + ret =csi_set_link_owner (csi,CSI_LINK_HOST ); +if (ret <0 ) + goto err; + +/* configure CSI-2 link */ + ret =csi_set_link_cfg (csi); +if (ret <0 ) + goto err; +
drivers/media/pci/intel/ivsc/mei_csi.c line 332
Quoted 10 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +10
-0+ ret =v4l2_subdev_call (csi->remote, video, s_stream,1 ); +if (ret) + goto err; + }else if (!enable && csi->streaming ==1 ) { +v4l2_subdev_call (csi->remote, video, s_stream,0 ); + +/* switch CSI-2 link to IVSC */ + ret =csi_set_link_owner (csi,CSI_LINK_IVSC ); +if (ret <0 ) + goto err;
I'd complain (e.g. dev_warn()) but return zero in this case. There's not much if anything the caller can do with this.
Quoted 52 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +52
-0+ } + + csi->streaming = enable; + +err: +return ret; +} + +static struct v4l2_mbus_framefmt * +mei_csi_get_pad_format (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +unsigned int pad,u32 which) +{ +struct mei_csi *csi =sd_to_csi (sd); + +switch (which) { +case V4L2_SUBDEV_FORMAT_TRY : +return v4l2_subdev_get_try_format (sd, sd_state, pad); +case V4L2_SUBDEV_FORMAT_ACTIVE : +return &csi->format_mbus[pad]; +default : +return NULL ; + } +} + +static int mei_csi_init_cfg (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state) +{ +struct v4l2_mbus_framefmt *mbusformat; +struct mei_csi *csi =sd_to_csi (sd); +unsigned int i; + +mutex_lock (&csi->lock); + +for (i =0 ; i < sd->entity.num_pads; i++) { + mbusformat =v4l2_subdev_get_try_format (sd, sd_state, i); + *mbusformat = mei_csi_format_mbus_default; + } + +mutex_unlock (&csi->lock); + +return 0 ; +} + +static int mei_csi_get_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *format) +{ +struct v4l2_mbus_framefmt *mbusformat; +struct mei_csi *csi =sd_to_csi (sd); + +mutex_lock (&csi->lock);
drivers/media/pci/intel/ivsc/mei_csi.c line 394
Quoted 10 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +10
-0+ + mbusformat =mei_csi_get_pad_format (sd, sd_state, format->pad, + format->which); +if (mbusformat) + format->format = *mbusformat; + +mutex_unlock (&csi->lock); + +return 0 ; +}
It'd be nice to have also enum_mbus_code support. Video mux of course didn't have this either. Something that could be done in the framework, including validating the mbus format.
Quoted 191 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +191
-0+ +static int mei_csi_set_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *format) +{ +struct v4l2_mbus_framefmt *source_mbusformat; +struct v4l2_mbus_framefmt *mbusformat; +struct mei_csi *csi =sd_to_csi (sd); +struct media_pad *pad; + + mbusformat =mei_csi_get_pad_format (sd, sd_state, format->pad, + format->which); +if (!mbusformat) +return -EINVAL ; + + source_mbusformat =mei_csi_get_pad_format (sd, sd_state, +CSI_PAD_SOURCE , + format->which); +if (!source_mbusformat) +return -EINVAL ; + +v4l_bound_align_image (&format->format.width,1 ,65536 ,0 , + &format->format.height,1 ,65536 ,0 ,0 ); + +switch (format->format.code) { +case MEDIA_BUS_FMT_RGB444_1X12 : +case MEDIA_BUS_FMT_RGB444_2X8_PADHI_BE : +case MEDIA_BUS_FMT_RGB444_2X8_PADHI_LE : +case MEDIA_BUS_FMT_RGB555_2X8_PADHI_BE : +case MEDIA_BUS_FMT_RGB555_2X8_PADHI_LE : +case MEDIA_BUS_FMT_RGB565_1X16 : +case MEDIA_BUS_FMT_BGR565_2X8_BE : +case MEDIA_BUS_FMT_BGR565_2X8_LE : +case MEDIA_BUS_FMT_RGB565_2X8_BE : +case MEDIA_BUS_FMT_RGB565_2X8_LE : +case MEDIA_BUS_FMT_RGB666_1X18 : +case MEDIA_BUS_FMT_RBG888_1X24 : +case MEDIA_BUS_FMT_RGB666_1X24_CPADHI : +case MEDIA_BUS_FMT_BGR888_1X24 : +case MEDIA_BUS_FMT_GBR888_1X24 : +case MEDIA_BUS_FMT_RGB888_1X24 : +case MEDIA_BUS_FMT_RGB888_2X12_BE : +case MEDIA_BUS_FMT_RGB888_2X12_LE : +case MEDIA_BUS_FMT_ARGB8888_1X32 : +case MEDIA_BUS_FMT_RGB888_1X32_PADHI : +case MEDIA_BUS_FMT_RGB101010_1X30 : +case MEDIA_BUS_FMT_RGB121212_1X36 : +case MEDIA_BUS_FMT_RGB161616_1X48 : +case MEDIA_BUS_FMT_Y8_1X8 : +case MEDIA_BUS_FMT_UV8_1X8 : +case MEDIA_BUS_FMT_UYVY8_1_5X8 : +case MEDIA_BUS_FMT_VYUY8_1_5X8 : +case MEDIA_BUS_FMT_YUYV8_1_5X8 : +case MEDIA_BUS_FMT_YVYU8_1_5X8 : +case MEDIA_BUS_FMT_UYVY8_2X8 : +case MEDIA_BUS_FMT_VYUY8_2X8 : +case MEDIA_BUS_FMT_YUYV8_2X8 : +case MEDIA_BUS_FMT_YVYU8_2X8 : +case MEDIA_BUS_FMT_Y10_1X10 : +case MEDIA_BUS_FMT_UYVY10_2X10 : +case MEDIA_BUS_FMT_VYUY10_2X10 : +case MEDIA_BUS_FMT_YUYV10_2X10 : +case MEDIA_BUS_FMT_YVYU10_2X10 : +case MEDIA_BUS_FMT_Y12_1X12 : +case MEDIA_BUS_FMT_UYVY12_2X12 : +case MEDIA_BUS_FMT_VYUY12_2X12 : +case MEDIA_BUS_FMT_YUYV12_2X12 : +case MEDIA_BUS_FMT_YVYU12_2X12 : +case MEDIA_BUS_FMT_UYVY8_1X16 : +case MEDIA_BUS_FMT_VYUY8_1X16 : +case MEDIA_BUS_FMT_YUYV8_1X16 : +case MEDIA_BUS_FMT_YVYU8_1X16 : +case MEDIA_BUS_FMT_YDYUYDYV8_1X16 : +case MEDIA_BUS_FMT_UYVY10_1X20 : +case MEDIA_BUS_FMT_VYUY10_1X20 : +case MEDIA_BUS_FMT_YUYV10_1X20 : +case MEDIA_BUS_FMT_YVYU10_1X20 : +case MEDIA_BUS_FMT_VUY8_1X24 : +case MEDIA_BUS_FMT_YUV8_1X24 : +case MEDIA_BUS_FMT_UYYVYY8_0_5X24 : +case MEDIA_BUS_FMT_UYVY12_1X24 : +case MEDIA_BUS_FMT_VYUY12_1X24 : +case MEDIA_BUS_FMT_YUYV12_1X24 : +case MEDIA_BUS_FMT_YVYU12_1X24 : +case MEDIA_BUS_FMT_YUV10_1X30 : +case MEDIA_BUS_FMT_UYYVYY10_0_5X30 : +case MEDIA_BUS_FMT_AYUV8_1X32 : +case MEDIA_BUS_FMT_UYYVYY12_0_5X36 : +case MEDIA_BUS_FMT_YUV12_1X36 : +case MEDIA_BUS_FMT_YUV16_1X48 : +case MEDIA_BUS_FMT_UYYVYY16_0_5X48 : +case MEDIA_BUS_FMT_JPEG_1X8 : +case MEDIA_BUS_FMT_AHSV8888_1X32 : +case MEDIA_BUS_FMT_SBGGR8_1X8 : +case MEDIA_BUS_FMT_SGBRG8_1X8 : +case MEDIA_BUS_FMT_SGRBG8_1X8 : +case MEDIA_BUS_FMT_SRGGB8_1X8 : +case MEDIA_BUS_FMT_SBGGR10_1X10 : +case MEDIA_BUS_FMT_SGBRG10_1X10 : +case MEDIA_BUS_FMT_SGRBG10_1X10 : +case MEDIA_BUS_FMT_SRGGB10_1X10 : +case MEDIA_BUS_FMT_SBGGR12_1X12 : +case MEDIA_BUS_FMT_SGBRG12_1X12 : +case MEDIA_BUS_FMT_SGRBG12_1X12 : +case MEDIA_BUS_FMT_SRGGB12_1X12 : +case MEDIA_BUS_FMT_SBGGR14_1X14 : +case MEDIA_BUS_FMT_SGBRG14_1X14 : +case MEDIA_BUS_FMT_SGRBG14_1X14 : +case MEDIA_BUS_FMT_SRGGB14_1X14 : +case MEDIA_BUS_FMT_SBGGR16_1X16 : +case MEDIA_BUS_FMT_SGBRG16_1X16 : +case MEDIA_BUS_FMT_SGRBG16_1X16 : +case MEDIA_BUS_FMT_SRGGB16_1X16 : +break ; +default : + format->format.code =MEDIA_BUS_FMT_Y8_1X8 ; +break ; + } +if (format->format.field ==V4L2_FIELD_ANY ) + format->format.field =V4L2_FIELD_NONE ; + +mutex_lock (&csi->lock); + + pad = &csi->pads[format->pad]; +if (pad->flags &MEDIA_PAD_FL_SOURCE ) + format->format = csi->format_mbus[CSI_PAD_SINK ]; + + *mbusformat = format->format; + +if (pad->flags &MEDIA_PAD_FL_SINK ) + *source_mbusformat = format->format; + +mutex_unlock (&csi->lock); + +return 0 ; +} + +static const struct v4l2_subdev_video_ops mei_csi_video_ops = { + .s_stream = mei_csi_set_stream, + .pre_streamon = mei_csi_pre_streamon, +}; + +static const struct v4l2_subdev_pad_ops mei_csi_pad_ops = { + .init_cfg = mei_csi_init_cfg, + .get_fmt = mei_csi_get_fmt, + .set_fmt = mei_csi_set_fmt, +}; + +static const struct v4l2_subdev_ops mei_csi_subdev_ops = { + .video = &mei_csi_video_ops, + .pad = &mei_csi_pad_ops, +}; + +static const struct media_entity_operations mei_csi_entity_ops = { + .link_validate = v4l2_subdev_link_validate, +}; + +static int mei_csi_notify_bound (struct v4l2_async_notifier *notifier, +struct v4l2_subdev *subdev, +struct v4l2_async_subdev *asd) +{ +struct mei_csi *csi =notifier_to_csi (notifier); +int pad; + + pad =media_entity_get_fwnode_pad (&subdev->entity, asd->match.fwnode, +MEDIA_PAD_FL_SOURCE ); +if (pad <0 ) +return pad; + + csi->remote = subdev; + csi->remote_pad = pad; + +return media_create_pad_link (&subdev->entity, pad, + &csi->subdev.entity,0 , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); +} + +static void mei_csi_notify_unbind (struct v4l2_async_notifier *notifier, +struct v4l2_subdev *subdev, +struct v4l2_async_subdev *asd) +{ +struct mei_csi *csi =notifier_to_csi (notifier); + + csi->remote =NULL ; +} + +static const struct v4l2_async_notifier_operations mei_csi_notify_ops = { + .bound = mei_csi_notify_bound, + .unbind = mei_csi_notify_unbind, +};
drivers/media/pci/intel/ivsc/mei_csi.c line 595
Quoted 10 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +10
-0+ +static int mei_csi_init_control (struct mei_csi *csi) +{ +v4l2_ctrl_handler_init (&csi->ctrl_handler,1 ); + csi->ctrl_handler.lock = &csi->lock; + + csi->privacy_ctrl =v4l2_ctrl_new_std (&csi->ctrl_handler,NULL , +V4L2_CID_PRIVACY ,0 ,1 ,1 ,0 ); +if (csi->ctrl_handler.error) +return csi->ctrl_handler.error;
The control should be made volatile here (flags |= V4L2_CTRL_FLAG_READ_ONLY) as it can't be changed by the user.
Quoted 89 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +89
-0+ + csi->subdev.ctrl_handler = &csi->ctrl_handler; + +return 0 ; +} + +static int mei_csi_parse_firmware (struct mei_csi *csi) +{ +struct v4l2_fwnode_endpoint v4l2_ep = { + .bus_type =V4L2_MBUS_CSI2_DPHY , + }; +struct device *dev = &csi->cldev->dev; +struct v4l2_async_subdev *asd; +struct fwnode_handle *fwnode; +struct fwnode_handle *ep; +int ret; + + ep =fwnode_graph_get_endpoint_by_id (dev_fwnode (dev),0 ,0 ,0 ); +if (!ep) { +dev_err (dev,"not connected to subdevice\n" ); +return -EINVAL ; + } + + ret =v4l2_fwnode_endpoint_parse (ep, &v4l2_ep); +if (ret) { +dev_err (dev,"could not parse v4l2 endpoint\n" ); +fwnode_handle_put (ep); +return -EINVAL ; + } + + fwnode =fwnode_graph_get_remote_endpoint (ep); +fwnode_handle_put (ep); + +v4l2_async_nf_init (&csi->notifier); + csi->notifier.ops = &mei_csi_notify_ops; + + asd =v4l2_async_nf_add_fwnode (&csi->notifier, fwnode, + struct v4l2_async_subdev); +fwnode_handle_put (fwnode); +if (IS_ERR (asd)) +return PTR_ERR (asd); + + ret =v4l2_async_subdev_nf_register (&csi->subdev, &csi->notifier); +if (ret) +v4l2_async_nf_cleanup (&csi->notifier); + +return ret; +} + +static int mei_csi_probe (struct mei_cl_device *cldev, +const struct mei_cl_device_id *id) +{ +struct mei_csi *csi; +int ret; + +/* +* no connections to sensor are defined in firmware, try to +* build connections as software_nodes parsed from SSDB. +*/ + ret =csi_bridge_init (cldev); +if (ret) +return ret; + + csi =devm_kzalloc (&cldev->dev,sizeof (struct mei_csi ),GFP_KERNEL ); +if (!csi) +return -ENOMEM ; + + csi->cldev = cldev; +mutex_init (&csi->lock); +init_completion (&csi->cmd_completion); + +mei_cldev_set_drvdata (cldev, csi); + + ret =mei_cldev_enable (cldev); +if (ret <0 ) { +dev_err (&cldev->dev,"mei_cldev_enable failed: %d\n" , ret); + goto destroy_mutex; + } + + ret =mei_cldev_register_rx_cb (cldev, mei_csi_rx); +if (ret) { +dev_err (&cldev->dev,"event cb registration failed: %d\n" , ret); + goto err_disable; + } + + ret =mei_csi_parse_firmware (csi); +if (ret) + goto err_disable; +
drivers/media/pci/intel/ivsc/mei_csi.c line 694
Quoted 10 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +10
-0+ csi->subdev.dev = &cldev->dev; +v4l2_subdev_init (&csi->subdev, &mei_csi_subdev_ops); +v4l2_set_subdevdata (&csi->subdev, csi); + csi->subdev.flags =V4L2_SUBDEV_FL_HAS_DEVNODE ; + csi->subdev.entity.function =MEDIA_ENT_F_VID_IF_BRIDGE ; + csi->subdev.entity.ops = &mei_csi_entity_ops; + + ret =mei_csi_init_control (csi); +if (ret) + goto err_async;
You'll also need to call v4l2_ctrl_handler_free() if setting up the handler failed. I.e. just change the label here and remove err_async label below.
Quoted 24 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +24
-0+ + csi->format_mbus[CSI_PAD_SOURCE ] = mei_csi_format_mbus_default; + csi->format_mbus[CSI_PAD_SINK ] = mei_csi_format_mbus_default; + + csi->pads[CSI_PAD_SOURCE ].flags =MEDIA_PAD_FL_SOURCE ; + csi->pads[CSI_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; + ret =media_entity_pads_init (&csi->subdev.entity,CSI_NUM_PADS , + csi->pads); +if (ret) + goto err_ctrl_handler; + + ret =v4l2_subdev_init_finalize (&csi->subdev); +if (ret <0 ) + goto err_entity; + + ret =v4l2_async_register_subdev (&csi->subdev); +if (ret <0 ) + goto err_subdev; + +pm_runtime_enable (&cldev->dev); + +return 0 ; + +err_subdev:
drivers/media/pci/intel/ivsc/mei_csi.c line 728
Quoted 10 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +10
-0+v4l2_subdev_cleanup (&csi->subdev); +err_entity: +media_entity_cleanup (&csi->subdev.entity); +err_ctrl_handler: +v4l2_ctrl_handler_free (&csi->ctrl_handler); +err_async: +v4l2_async_nf_unregister (&csi->notifier); +v4l2_async_nf_cleanup (&csi->notifier); +err_disable: +mei_cldev_disable (cldev);
It'd be nice to have an additional newline before the label. Elsewhere, too.
Quoted 18 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +18
-0+destroy_mutex: +mutex_destroy (&csi->lock); + +return ret; +} + +static void mei_csi_remove (struct mei_cl_device *cldev) +{ +struct mei_csi *csi =mei_cldev_get_drvdata (cldev); + +v4l2_async_nf_unregister (&csi->notifier); +v4l2_async_nf_cleanup (&csi->notifier); +v4l2_ctrl_handler_free (&csi->ctrl_handler); +v4l2_async_unregister_subdev (&csi->subdev); +v4l2_subdev_cleanup (&csi->subdev); +media_entity_cleanup (&csi->subdev.entity); + +pm_runtime_disable (&cldev->dev);
drivers/media/pci/intel/ivsc/mei_csi.c line 756
Quoted 10 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +10
-0+ +mutex_destroy (&csi->lock); +} + +#define MEI_CSI_UUID UUID_LE(0x92335FCF, 0x3203, 0x4472, \ + 0xAF, 0x93, 0x7b, 0x44, 0x53, 0xAC, 0x29, 0xDA) + +static const struct mei_cl_device_id mei_csi_tbl[] = { + {MEI_CSI_DRIVER_NAME ,MEI_CSI_UUID ,MEI_CL_VERSION_ANY }, +
Extra newline.
drivers/media/pci/intel/ivsc/mei_csi.c line 757
Quoted 1 line
drivers/media/pci/intel/ivsc/mei_csi.c new file +1
-0+/* required last entry */
The comment could be on the same line below. E.g. "Sentinel" is appropriate, as this isn't just any last entry.
Quoted 18 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +18
-0+ { } +}; +MODULE_DEVICE_TABLE (mei, mei_csi_tbl); + +static struct mei_cl_driver mei_csi_driver = { + .id_table = mei_csi_tbl, + .name =MEI_CSI_DRIVER_NAME , + + .probe = mei_csi_probe, + .remove = mei_csi_remove, +}; + +module_mei_cl_driver (mei_csi_driver); + +MODULE_AUTHOR ("Wentong Wu <[email protected]>" ); +MODULE_AUTHOR ("Zhifeng Wang <[email protected]>" ); +MODULE_DESCRIPTION ("Device driver for IVSC CSI" ); +MODULE_LICENSE ("GPL" );
Signature · 3 lines
Kind regards, Sakari Ailus
RE: [PATCH v3 1/3] media: pci: intel: ivsc: Add CSI submodule
Hi Sakari,
Thanks
Quoted 241 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Wednesday, March 29, 2023 4:49 PMHi Wentong,
Quoted 17 lines · Wentong Wu
CSI is a submodule of IVSC which can route camera sensor data to the outbound MIPI CSI-2 interface.
The interface communicating with firmware is via MEI. There is a separate MEI UUID, which this driver uses to enumerate.
To route camera sensor data to host, the information of link frequency and number of data lanes is sent to firmware by sending MEI command when starting stream.
CSI also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both IVSC and host Image Processing Unit(IPU) can't get image data. And when this mode is turned on, host Image Processing Unit(IPU) driver is informed via v4l2 control callback, so that user can be notified.
Signed-off-by Wentong Wu <[email protected]>Quoted 8 lines
--- drivers/media/pci/Kconfig | 1 + drivers/media/pci/intel/Makefile | 2 + drivers/media/pci/intel/ivsc/Kconfig | 12 + drivers/media/pci/intel/ivsc/Makefile | 7 + drivers/media/pci/intel/ivsc/csi_bridge.c | 332 +++++++++++++ drivers/media/pci/intel/ivsc/csi_bridge.h | 122 +++++ drivers/media/pci/intel/ivsc/mei_csi.c | 775++++++++++++++++++++++++++++++
Quoted 6 lines
7 files changed, 1251 insertions(+) create mode 100644 drivers/media/pci/intel/ivsc/Kconfig create mode 100644 drivers/media/pci/intel/ivsc/Makefile create mode 100644 drivers/media/pci/intel/ivsc/csi_bridge.c create mode 100644 drivers/media/pci/intel/ivsc/csi_bridge.h create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.cI'm commenting just
mei_csithis time...Quoted 199 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +192
-0@@ -0,0 +1,775 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright (C) 2023 Intel Corporation. All rights reserved. +* Intel Visual Sensing Controller CSI Linux driver */ + +/* +* To set ownership of CSI-2 link and to configure CSI-2 link, there +* are specific commands, which are sent via MEI protocol. The send +* command function uses "completion" as a synchronization mechanism. +* The response for command is received via a mei callback which +wakes +* up the caller. There can be only one outstanding command at a time. +*/ + +#include <linux/completion.h> +#include <linux/delay.h> +#include <linux/kernel.h> +#include <linux/math64.h> +#include <linux/mei_cl_bus.h> +#include <linux/module.h> +#include <linux/mutex.h> +#include <linux/pm_runtime.h> +#include <linux/slab.h> +#include <linux/units.h> +#include <linux/uuid.h> +#include <linux/workqueue.h> + +#include <media/v4l2-async.h> +#include <media/v4l2-ctrls.h> +#include <media/v4l2-fwnode.h> +#include <media/v4l2-subdev.h> + +#define MEI_CSI_DRIVER_NAME "ivsc_csi" + +/* the 5s used here is based on experiment */ #define CSI_CMD_TIMEOUT +(5 *HZ ) +/* to setup CSI-2 link an extra delay needed and determined +experimentally */ #define CSI_FW_READY_DELAY_MS 100 +/* link frequency unit is 100kHz */ +#define CSI_LINK_FREQ (x) ((u32)(div_u64(x, 100 * HZ_PER_KHZ))) + +/* +* identify the command id supported by firmware +* IPC, as well as the privacy notification id +* used when processing privacy event. +*/ +enum csi_cmd_id { +/* used to set csi ownership */ +CSI_SET_OWNER =0 , + +/* used to configure CSI-2 link */ +CSI_SET_CONF =2 , + +/* privacy notification id used when privacy state changes */ +CSI_PRIVACY_NOTIF =6 , +}; + +/* CSI-2 link ownership definition */ enum csi_link_owner { +CSI_LINK_IVSC , +CSI_LINK_HOST , +}; + +/* privacy status definition */ +enum ivsc_privacy_status { +CSI_PRIVACY_OFF , +CSI_PRIVACY_ON , +CSI_PRIVACY_MAX , +}; + +enum csi_pads { +CSI_PAD_SOURCE , +CSI_PAD_SINK , +CSI_NUM_PADS +}; + +/* configuration of the CSI-2 link between host and IVSC */ struct +csi_link_cfg { +/* number of data lanes used on the CSI-2 link */ +u32 nr_of_lanes; + +/* frequency of the CSI-2 link */ +u32 link_freq; + +/* for future use */ +u32 rsvd[2 ]; +} __packed; + +/* CSI command structure */ +struct csi_cmd { +u32 cmd_id; +union _cmd_param { +u32 param; +struct csi_link_cfg conf; + } param; +} __packed; + +/* CSI notification structure */ +struct csi_notif { +u32 cmd_id; +int status; +union _resp_cont { +u32 cont; +struct csi_link_cfg conf; + } cont; +} __packed; + +struct mei_csi { +struct mei_cl_device *cldev; + +/* command response */ +struct csi_notif cmd_response; +/* used to wait for command response from firmware */ +struct completion cmd_completion; +/* protect command download */ +struct mutex lock; + +struct v4l2_subdev subdev; +struct v4l2_subdev *remote; +struct v4l2_async_notifier notifier; +struct v4l2_ctrl_handler ctrl_handler; +struct v4l2_ctrl *privacy_ctrl; +unsigned int remote_pad; +/* start streaming or not */ +int streaming; + +struct media_pad pads[CSI_NUM_PADS ]; +struct v4l2_mbus_framefmt format_mbus[CSI_NUM_PADS ]; + +/* number of data lanes used on the CSI-2 link */ +u32 nr_of_lanes; +/* frequency of the CSI-2 link */ +u64 link_freq; + +/* privacy status */ +enum ivsc_privacy_status status; +}; + +static const struct v4l2_mbus_framefmt mei_csi_format_mbus_default = { + .width =1 , + .height =1 , + .code =MEDIA_BUS_FMT_Y8_1X8 , + .field =V4L2_FIELD_NONE , +}; + +int csi_bridge_init (struct mei_cl_device *csi_dev); + +static inline struct mei_csi *notifier_to_csi (struct +v4l2_async_notifier *n) { + return container_of(n, struct mei_csi, notifier); } + +static inline struct mei_csi *sd_to_csi (struct v4l2_subdev *sd) { + return container_of(sd, struct mei_csi, subdev); } + +/* send a command to firmware and mutex must be held by caller */ +static int mei_csi_send (struct mei_csi *csi,u8 *buf,size_t len) { +struct csi_cmd *cmd = (struct csi_cmd *)buf; +int ret; + +reinit_completion (&csi->cmd_completion); + + ret =mei_cldev_send (csi->cldev, buf, len); +if (ret <0 ) + goto out; + + ret =wait_for_completion_killable_timeout (&csi->cmd_completion, +CSI_CMD_TIMEOUT ); +if (ret <0 ) { + goto out; + }else if (!ret) { + ret = -ETIMEDOUT ; + goto out; + } + +/* command response status */ + ret = csi->cmd_response.status; +if (ret) { + ret = -EINVAL ; + goto out; + } + +if (csi->cmd_response.cmd_id != cmd->cmd_id) + ret = -EINVAL ; + +out: +return ret; +} + +/* set CSI-2 link ownership */ +static int csi_set_link_owner (struct mei_csi *csi,enum +csi_link_owner owner) { +struct csi_cmd cmd = {0 }; +size_t cmd_size;
Quoted 8 lines
drivers/media/pci/intel/ivsc/mei_csi.c line 202
Quoted 5 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +5
-0+int ret; + + cmd.cmd_id =CSI_SET_OWNER ; + cmd.param.param = owner; + cmd_size =sizeof (cmd.cmd_id) +sizeof (cmd.param.param);In some cases you're using memset and in others not. If you don't need memset, I'd prefer assigning the fields in variable declaration instead.
The declaration will be like below, but it will break reverse x-mas tree style.
struct csi_cmd cmd = { 0 };size_t cmd_size = sizeof(cmd.cmd_id) + sizeof(cmd.param.param); int ret;
Quoted 80 lines
Quoted 11 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +10
-0+ +mutex_lock (&csi->lock); + + ret =mei_csi_send (csi, (u8 *)&cmd, cmd_size); + +mutex_unlock (&csi->lock); + +return ret; +} +drivers/media/pci/intel/ivsc/mei_csi.c line 223
Quoted 10 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +10
-0+/* configure CSI-2 link between host and IVSC */ static int +csi_set_link_cfg (struct mei_csi *csi) { +struct csi_cmd cmd = {0 }; +size_t cmd_size; +int ret; + + cmd.cmd_id =CSI_SET_CONF ; + cmd.param.conf.nr_of_lanes = csi->nr_of_lanes; + cmd.param.conf.link_freq =CSI_LINK_FREQ (csi->link_freq); + cmd_size =sizeof (cmd.cmd_id) +sizeof (cmd.param.conf);Ditto.
Quoted 56 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +56
-0+ +mutex_lock (&csi->lock); + + ret =mei_csi_send (csi, (u8 *)&cmd, cmd_size); +/* +* wait configuration ready if download success. placing +* delay under mutex is to make sure current command flow +* completed before starting a possible new one. +*/ +if (!ret) +msleep (CSI_FW_READY_DELAY_MS ); + +mutex_unlock (&csi->lock); + +return ret; +} + +/* callback for receive */ +static void mei_csi_rx (struct mei_cl_device *cldev) { +struct mei_csi *csi =mei_cldev_get_drvdata (cldev); +struct csi_notif notif = {0 }; +int ret; + + ret =mei_cldev_recv (cldev, (u8 *)¬if,sizeof (notif)); +if (ret <0 ) { +dev_err (&cldev->dev,"recv error: %d\n" , ret); +return ; + } + +switch (notif.cmd_id) { +case CSI_PRIVACY_NOTIF : +if (notif.cont.cont <CSI_PRIVACY_MAX ) { + csi->status = notif.cont.cont; +v4l2_ctrl_s_ctrl (csi->privacy_ctrl, csi->status); + } +break ; +case CSI_SET_OWNER : +case CSI_SET_CONF : +memcpy (&csi->cmd_response, ¬if, ret); + +complete (&csi->cmd_completion); +break ; +default : +break ; + } +} + +static int mei_csi_pre_streamon (struct v4l2_subdev *sd,u32 flags) { +struct v4l2_querymenu qm = { .id =V4L2_CID_LINK_FREQ , }; +struct v4l2_mbus_config mbus_config = {0 }; +struct mei_csi *csi =sd_to_csi (sd); +struct v4l2_ctrl *ctrl; +int ret =0 ; + +if (!csi->remote) +return -ENODEV ;
Quoted 8 lines
drivers/media/pci/intel/ivsc/mei_csi.c line 286
Quoted 5 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +5
-0+ + ret =v4l2_subdev_call (csi->remote, pad, get_mbus_config, + csi->remote_pad, &mbus_config); +if (ret) +return ret;You're already parsing the endpoint in
probe(). Do you need this, assuming the sensor has a static lane configuration? Virtually all do.
When parsing remote endpoint, just get num_data_lanes by v4l2_fwnode_endpoint_alloc_parse?
Quoted 2 lines
Could this all be done via the
s_stream()op instead?
We don't need csi2 link freq and lane number when s_steam(0)
Quoted 41 lines
Quoted 41 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +40
-0+ +if (mbus_config.type !=V4L2_MBUS_CSI2_DPHY ) +return -EINVAL ; + + csi->nr_of_lanes = mbus_config.bus.mipi_csi2.num_data_lanes; + + ctrl =v4l2_ctrl_find (csi->remote->ctrl_handler,V4L2_CID_LINK_FREQ ); +if (!ctrl) +return -EINVAL ; + qm.index =v4l2_ctrl_g_ctrl (ctrl); + + ret =v4l2_querymenu (csi->remote->ctrl_handler, &qm); +if (ret) +return ret; + + csi->link_freq = qm.value; + +return ret; +} + +static int mei_csi_set_stream (struct v4l2_subdev *sd,int enable) { +struct mei_csi *csi =sd_to_csi (sd); +int ret =0 ; + +if (enable && csi->streaming ==0 ) { +/* switch CSI-2 link to host */ + ret =csi_set_link_owner (csi,CSI_LINK_HOST ); +if (ret <0 ) + goto err; + +/* configure CSI-2 link */ + ret =csi_set_link_cfg (csi); +if (ret <0 ) + goto err; + + ret =v4l2_subdev_call (csi->remote, video, s_stream,1 ); +if (ret) + goto err; + }else if (!enable && csi->streaming ==1 ) { +v4l2_subdev_call (csi->remote, video, s_stream,0 );
Quoted 8 lines
drivers/media/pci/intel/ivsc/mei_csi.c line 332
Quoted 5 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +5
-0+ +/* switch CSI-2 link to IVSC */ + ret =csi_set_link_owner (csi,CSI_LINK_IVSC ); +if (ret <0 ) + goto err;I'd complain (e.g.
dev_warn()) but return zero in this case. There's not much if anything the caller can do with this.
ack
Quoted 56 lines
Quoted 56 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +55
-0+ } + + csi->streaming = enable; + +err: +return ret; +} + +static struct v4l2_mbus_framefmt * +mei_csi_get_pad_format (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +unsigned int pad,u32 which) { +struct mei_csi *csi =sd_to_csi (sd); + +switch (which) { +case V4L2_SUBDEV_FORMAT_TRY : +return v4l2_subdev_get_try_format (sd, sd_state, pad); +case V4L2_SUBDEV_FORMAT_ACTIVE : +return &csi->format_mbus[pad]; +default : +return NULL ; + } +} + +static int mei_csi_init_cfg (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state) { +struct v4l2_mbus_framefmt *mbusformat; +struct mei_csi *csi =sd_to_csi (sd); +unsigned int i; + +mutex_lock (&csi->lock); + +for (i =0 ; i < sd->entity.num_pads; i++) { + mbusformat =v4l2_subdev_get_try_format (sd, sd_state, i); + *mbusformat = mei_csi_format_mbus_default; + } + +mutex_unlock (&csi->lock); + +return 0 ; +} + +static int mei_csi_get_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *format) { +struct v4l2_mbus_framefmt *mbusformat; +struct mei_csi *csi =sd_to_csi (sd); + +mutex_lock (&csi->lock); + + mbusformat =mei_csi_get_pad_format (sd, sd_state, format->pad, + format->which); +if (mbusformat) + format->format = *mbusformat; +
Quoted 8 lines
drivers/media/pci/intel/ivsc/mei_csi.c line 375
Quoted 4 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +4
-0+mutex_unlock (&csi->lock); + +return 0 ; +}It'd be nice to have also
enum_mbus_codesupport. Video mux of course didn't have this either. Something that could be done in the framework, including validating the mbus format.
Sorry, you mean the default MEDIA_BUS_FMT_Y8_1X8?
Quoted 194 lines
Quoted 194 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +162
-0+ +static int mei_csi_set_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *format) { +struct v4l2_mbus_framefmt *source_mbusformat; +struct v4l2_mbus_framefmt *mbusformat; +struct mei_csi *csi =sd_to_csi (sd); +struct media_pad *pad; + + mbusformat =mei_csi_get_pad_format (sd, sd_state, format->pad, + format->which); +if (!mbusformat) +return -EINVAL ; + + source_mbusformat =mei_csi_get_pad_format (sd, sd_state, +CSI_PAD_SOURCE , + format->which); +if (!source_mbusformat) +return -EINVAL ; + +v4l_bound_align_image (&format->format.width,1 ,65536 ,0 , + &format->format.height,1 ,65536 ,0 ,0 ); + +switch (format->format.code) { +case MEDIA_BUS_FMT_RGB444_1X12 : +case MEDIA_BUS_FMT_RGB444_2X8_PADHI_BE : +case MEDIA_BUS_FMT_RGB444_2X8_PADHI_LE : +case MEDIA_BUS_FMT_RGB555_2X8_PADHI_BE : +case MEDIA_BUS_FMT_RGB555_2X8_PADHI_LE : +case MEDIA_BUS_FMT_RGB565_1X16 : +case MEDIA_BUS_FMT_BGR565_2X8_BE : +case MEDIA_BUS_FMT_BGR565_2X8_LE : +case MEDIA_BUS_FMT_RGB565_2X8_BE : +case MEDIA_BUS_FMT_RGB565_2X8_LE : +case MEDIA_BUS_FMT_RGB666_1X18 : +case MEDIA_BUS_FMT_RBG888_1X24 : +case MEDIA_BUS_FMT_RGB666_1X24_CPADHI : +case MEDIA_BUS_FMT_BGR888_1X24 : +case MEDIA_BUS_FMT_GBR888_1X24 : +case MEDIA_BUS_FMT_RGB888_1X24 : +case MEDIA_BUS_FMT_RGB888_2X12_BE : +case MEDIA_BUS_FMT_RGB888_2X12_LE : +case MEDIA_BUS_FMT_ARGB8888_1X32 : +case MEDIA_BUS_FMT_RGB888_1X32_PADHI : +case MEDIA_BUS_FMT_RGB101010_1X30 : +case MEDIA_BUS_FMT_RGB121212_1X36 : +case MEDIA_BUS_FMT_RGB161616_1X48 : +case MEDIA_BUS_FMT_Y8_1X8 : +case MEDIA_BUS_FMT_UV8_1X8 : +case MEDIA_BUS_FMT_UYVY8_1_5X8 : +case MEDIA_BUS_FMT_VYUY8_1_5X8 : +case MEDIA_BUS_FMT_YUYV8_1_5X8 : +case MEDIA_BUS_FMT_YVYU8_1_5X8 : +case MEDIA_BUS_FMT_UYVY8_2X8 : +case MEDIA_BUS_FMT_VYUY8_2X8 : +case MEDIA_BUS_FMT_YUYV8_2X8 : +case MEDIA_BUS_FMT_YVYU8_2X8 : +case MEDIA_BUS_FMT_Y10_1X10 : +case MEDIA_BUS_FMT_UYVY10_2X10 : +case MEDIA_BUS_FMT_VYUY10_2X10 : +case MEDIA_BUS_FMT_YUYV10_2X10 : +case MEDIA_BUS_FMT_YVYU10_2X10 : +case MEDIA_BUS_FMT_Y12_1X12 : +case MEDIA_BUS_FMT_UYVY12_2X12 : +case MEDIA_BUS_FMT_VYUY12_2X12 : +case MEDIA_BUS_FMT_YUYV12_2X12 : +case MEDIA_BUS_FMT_YVYU12_2X12 : +case MEDIA_BUS_FMT_UYVY8_1X16 : +case MEDIA_BUS_FMT_VYUY8_1X16 : +case MEDIA_BUS_FMT_YUYV8_1X16 : +case MEDIA_BUS_FMT_YVYU8_1X16 : +case MEDIA_BUS_FMT_YDYUYDYV8_1X16 : +case MEDIA_BUS_FMT_UYVY10_1X20 : +case MEDIA_BUS_FMT_VYUY10_1X20 : +case MEDIA_BUS_FMT_YUYV10_1X20 : +case MEDIA_BUS_FMT_YVYU10_1X20 : +case MEDIA_BUS_FMT_VUY8_1X24 : +case MEDIA_BUS_FMT_YUV8_1X24 : +case MEDIA_BUS_FMT_UYYVYY8_0_5X24 : +case MEDIA_BUS_FMT_UYVY12_1X24 : +case MEDIA_BUS_FMT_VYUY12_1X24 : +case MEDIA_BUS_FMT_YUYV12_1X24 : +case MEDIA_BUS_FMT_YVYU12_1X24 : +case MEDIA_BUS_FMT_YUV10_1X30 : +case MEDIA_BUS_FMT_UYYVYY10_0_5X30 : +case MEDIA_BUS_FMT_AYUV8_1X32 : +case MEDIA_BUS_FMT_UYYVYY12_0_5X36 : +case MEDIA_BUS_FMT_YUV12_1X36 : +case MEDIA_BUS_FMT_YUV16_1X48 : +case MEDIA_BUS_FMT_UYYVYY16_0_5X48 : +case MEDIA_BUS_FMT_JPEG_1X8 : +case MEDIA_BUS_FMT_AHSV8888_1X32 : +case MEDIA_BUS_FMT_SBGGR8_1X8 : +case MEDIA_BUS_FMT_SGBRG8_1X8 : +case MEDIA_BUS_FMT_SGRBG8_1X8 : +case MEDIA_BUS_FMT_SRGGB8_1X8 : +case MEDIA_BUS_FMT_SBGGR10_1X10 : +case MEDIA_BUS_FMT_SGBRG10_1X10 : +case MEDIA_BUS_FMT_SGRBG10_1X10 : +case MEDIA_BUS_FMT_SRGGB10_1X10 : +case MEDIA_BUS_FMT_SBGGR12_1X12 : +case MEDIA_BUS_FMT_SGBRG12_1X12 : +case MEDIA_BUS_FMT_SGRBG12_1X12 : +case MEDIA_BUS_FMT_SRGGB12_1X12 : +case MEDIA_BUS_FMT_SBGGR14_1X14 : +case MEDIA_BUS_FMT_SGBRG14_1X14 : +case MEDIA_BUS_FMT_SGRBG14_1X14 : +case MEDIA_BUS_FMT_SRGGB14_1X14 : +case MEDIA_BUS_FMT_SBGGR16_1X16 : +case MEDIA_BUS_FMT_SGBRG16_1X16 : +case MEDIA_BUS_FMT_SGRBG16_1X16 : +case MEDIA_BUS_FMT_SRGGB16_1X16 : +break ; +default : + format->format.code =MEDIA_BUS_FMT_Y8_1X8 ; +break ; + } +if (format->format.field ==V4L2_FIELD_ANY ) + format->format.field =V4L2_FIELD_NONE ; + +mutex_lock (&csi->lock); + + pad = &csi->pads[format->pad]; +if (pad->flags &MEDIA_PAD_FL_SOURCE ) + format->format = csi->format_mbus[CSI_PAD_SINK ]; + + *mbusformat = format->format; + +if (pad->flags &MEDIA_PAD_FL_SINK ) + *source_mbusformat = format->format; + +mutex_unlock (&csi->lock); + +return 0 ; +} + +static const struct v4l2_subdev_video_ops mei_csi_video_ops = { + .s_stream = mei_csi_set_stream, + .pre_streamon = mei_csi_pre_streamon, }; + +static const struct v4l2_subdev_pad_ops mei_csi_pad_ops = { + .init_cfg = mei_csi_init_cfg, + .get_fmt = mei_csi_get_fmt, + .set_fmt = mei_csi_set_fmt, +}; + +static const struct v4l2_subdev_ops mei_csi_subdev_ops = { + .video = &mei_csi_video_ops, + .pad = &mei_csi_pad_ops, +}; + +static const struct media_entity_operations mei_csi_entity_ops = { + .link_validate = v4l2_subdev_link_validate, }; + +static int mei_csi_notify_bound (struct v4l2_async_notifier *notifier, +struct v4l2_subdev *subdev, +struct v4l2_async_subdev *asd) +{ +struct mei_csi *csi =notifier_to_csi (notifier); +int pad; + + pad = media_entity_get_fwnode_pad(&subdev->entity, asd-match.fwnode,
drivers/media/pci/intel/ivsc/mei_csi.c new file +30
-0+ MEDIA_PAD_FL_SOURCE); +if (pad <0 ) +return pad; + + csi->remote = subdev; + csi->remote_pad = pad; + +return media_create_pad_link (&subdev->entity, pad, + &csi->subdev.entity,0 , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); +} + +static void mei_csi_notify_unbind (struct v4l2_async_notifier *notifier, +struct v4l2_subdev *subdev, +struct v4l2_async_subdev *asd) +{ +struct mei_csi *csi =notifier_to_csi (notifier); + + csi->remote =NULL ; +} + +static const struct v4l2_async_notifier_operations mei_csi_notify_ops = { + .bound = mei_csi_notify_bound, + .unbind = mei_csi_notify_unbind, +}; + +static int mei_csi_init_control (struct mei_csi *csi) { +v4l2_ctrl_handler_init (&csi->ctrl_handler,1 ); + csi->ctrl_handler.lock = &csi->lock;
Quoted 8 lines
drivers/media/pci/intel/ivsc/mei_csi.c line 595
Quoted 5 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +5
-0+ + csi->privacy_ctrl =v4l2_ctrl_new_std (&csi->ctrl_handler,NULL , +V4L2_CID_PRIVACY ,0 ,1 ,1 ,0 ); +if (csi->ctrl_handler.error) +return csi->ctrl_handler.error;The control should be made volatile here (flags |=
V4L2_CTRL_FLAG_READ_ONLY) as it can't be changed by the user.
ack
Quoted 93 lines
Quoted 93 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +92
-0+ + csi->subdev.ctrl_handler = &csi->ctrl_handler; + +return 0 ; +} + +static int mei_csi_parse_firmware (struct mei_csi *csi) { +struct v4l2_fwnode_endpoint v4l2_ep = { + .bus_type =V4L2_MBUS_CSI2_DPHY , + }; +struct device *dev = &csi->cldev->dev; +struct v4l2_async_subdev *asd; +struct fwnode_handle *fwnode; +struct fwnode_handle *ep; +int ret; + + ep =fwnode_graph_get_endpoint_by_id (dev_fwnode (dev),0 ,0 ,0 ); +if (!ep) { +dev_err (dev,"not connected to subdevice\n" ); +return -EINVAL ; + } + + ret =v4l2_fwnode_endpoint_parse (ep, &v4l2_ep); +if (ret) { +dev_err (dev,"could not parse v4l2 endpoint\n" ); +fwnode_handle_put (ep); +return -EINVAL ; + } + + fwnode =fwnode_graph_get_remote_endpoint (ep); +fwnode_handle_put (ep); + +v4l2_async_nf_init (&csi->notifier); + csi->notifier.ops = &mei_csi_notify_ops; + + asd =v4l2_async_nf_add_fwnode (&csi->notifier, fwnode, + struct v4l2_async_subdev); +fwnode_handle_put (fwnode); +if (IS_ERR (asd)) +return PTR_ERR (asd); + + ret =v4l2_async_subdev_nf_register (&csi->subdev, &csi->notifier); +if (ret) +v4l2_async_nf_cleanup (&csi->notifier); + +return ret; +} + +static int mei_csi_probe (struct mei_cl_device *cldev, +const struct mei_cl_device_id *id) { +struct mei_csi *csi; +int ret; + +/* +* no connections to sensor are defined in firmware, try to +* build connections as software_nodes parsed from SSDB. +*/ + ret =csi_bridge_init (cldev); +if (ret) +return ret; + + csi =devm_kzalloc (&cldev->dev,sizeof (struct mei_csi ),GFP_KERNEL ); +if (!csi) +return -ENOMEM ; + + csi->cldev = cldev; +mutex_init (&csi->lock); +init_completion (&csi->cmd_completion); + +mei_cldev_set_drvdata (cldev, csi); + + ret =mei_cldev_enable (cldev); +if (ret <0 ) { +dev_err (&cldev->dev,"mei_cldev_enable failed: %d\n" , ret); + goto destroy_mutex; + } + + ret =mei_cldev_register_rx_cb (cldev, mei_csi_rx); +if (ret) { +dev_err (&cldev->dev,"event cb registration failed: %d\n" , ret); + goto err_disable; + } + + ret =mei_csi_parse_firmware (csi); +if (ret) + goto err_disable; + + csi->subdev.dev = &cldev->dev; +v4l2_subdev_init (&csi->subdev, &mei_csi_subdev_ops); +v4l2_set_subdevdata (&csi->subdev, csi); + csi->subdev.flags =V4L2_SUBDEV_FL_HAS_DEVNODE ; + csi->subdev.entity.function =MEDIA_ENT_F_VID_IF_BRIDGE ;
Quoted 8 lines
drivers/media/pci/intel/ivsc/mei_csi.c line 694
Quoted 5 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +5
-0+ csi->subdev.entity.ops = &mei_csi_entity_ops; + + ret =mei_csi_init_control (csi); +if (ret) + goto err_async;You'll also need to call
v4l2_ctrl_handler_free()if setting up the handler failed. I.e. just change the label here and removeerr_asynclabel below.
Thanks
Quoted 29 lines
Quoted 29 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +28
-0+ + csi->format_mbus[CSI_PAD_SOURCE ] = mei_csi_format_mbus_default; + csi->format_mbus[CSI_PAD_SINK ] = mei_csi_format_mbus_default; + + csi->pads[CSI_PAD_SOURCE ].flags =MEDIA_PAD_FL_SOURCE ; + csi->pads[CSI_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; + ret =media_entity_pads_init (&csi->subdev.entity,CSI_NUM_PADS , + csi->pads); +if (ret) + goto err_ctrl_handler; + + ret =v4l2_subdev_init_finalize (&csi->subdev); +if (ret <0 ) + goto err_entity; + + ret =v4l2_async_register_subdev (&csi->subdev); +if (ret <0 ) + goto err_subdev; + +pm_runtime_enable (&cldev->dev); + +return 0 ; + +err_subdev: +v4l2_subdev_cleanup (&csi->subdev); +err_entity: +media_entity_cleanup (&csi->subdev.entity); +err_ctrl_handler:
Quoted 8 lines
drivers/media/pci/intel/ivsc/mei_csi.c line 728
Quoted 6 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +6
-0+v4l2_ctrl_handler_free (&csi->ctrl_handler); +err_async: +v4l2_async_nf_unregister (&csi->notifier); +v4l2_async_nf_cleanup (&csi->notifier); +err_disable: +mei_cldev_disable (cldev);It'd be nice to have an additional newline before the label. Elsewhere, too.
ack
Quoted 31 lines
Quoted 18 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +17
-0+destroy_mutex: +mutex_destroy (&csi->lock); + +return ret; +} + +static void mei_csi_remove (struct mei_cl_device *cldev) { +struct mei_csi *csi =mei_cldev_get_drvdata (cldev); + +v4l2_async_nf_unregister (&csi->notifier); +v4l2_async_nf_cleanup (&csi->notifier); +v4l2_ctrl_handler_free (&csi->ctrl_handler); +v4l2_async_unregister_subdev (&csi->subdev); +v4l2_subdev_cleanup (&csi->subdev); +media_entity_cleanup (&csi->subdev.entity); + +pm_runtime_disable (&cldev->dev);drivers/media/pci/intel/ivsc/mei_csi.c line 756
Quoted 10 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +10
-0+ +mutex_destroy (&csi->lock); +} + +#define MEI_CSI_UUID UUID_LE(0x92335FCF, 0x3203, 0x4472, \ + 0xAF, 0x93, 0x7b, 0x44, 0x53, 0xAC, 0x29, 0xDA) + +static const struct mei_cl_device_id mei_csi_tbl[] = { + {MEI_CSI_DRIVER_NAME ,MEI_CSI_UUID ,MEI_CL_VERSION_ANY }, +Extra newline.
Quoted 4 lines
drivers/media/pci/intel/ivsc/mei_csi.c line 757
Quoted 1 line
drivers/media/pci/intel/ivsc/mei_csi.c new file +1
-0+/* required last entry */ The comment could be on the same line below. E.g. "Sentinel" is appropriate, as this isn't just any last entry.
thanks
Quoted 24 lines
Quoted 9 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +8
-0+ { } +}; +MODULE_DEVICE_TABLE (mei, mei_csi_tbl); + +static struct mei_cl_driver mei_csi_driver = { + .id_table = mei_csi_tbl, + .name =MEI_CSI_DRIVER_NAME , +drivers/media/pci/intel/ivsc/mei_csi.c line 775
Quoted 10 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +10
-0+ .probe = mei_csi_probe, + .remove = mei_csi_remove, +}; + +module_mei_cl_driver (mei_csi_driver); + +MODULE_AUTHOR ("Wentong Wu <[email protected]>" ); +MODULE_AUTHOR ("Zhifeng Wang <[email protected]>" ); +MODULE_DESCRIPTION ("Device driver for IVSC CSI" ); +MODULE_LICENSE ("GPL" );--
Kind regards,Sakari Ailus
Sakari Ailus
maintainerRe: [PATCH v3 1/3] media: pci: intel: ivsc: Add CSI submodule
Hi Wentong,
Quoted 254 lines · Wentong
Hi Sakari,
Thanks
Quoted 241 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Wednesday, March 29, 2023 4:49 PMHi Wentong,
Quoted 17 lines · Wentong Wu
CSI is a submodule of IVSC which can route camera sensor data to the outbound MIPI CSI-2 interface.
The interface communicating with firmware is via MEI. There is a separate MEI UUID, which this driver uses to enumerate.
To route camera sensor data to host, the information of link frequency and number of data lanes is sent to firmware by sending MEI command when starting stream.
CSI also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both IVSC and host Image Processing Unit(IPU) can't get image data. And when this mode is turned on, host Image Processing Unit(IPU) driver is informed via v4l2 control callback, so that user can be notified.
Signed-off-by Wentong Wu <[email protected]>Quoted 8 lines
--- drivers/media/pci/Kconfig | 1 + drivers/media/pci/intel/Makefile | 2 + drivers/media/pci/intel/ivsc/Kconfig | 12 + drivers/media/pci/intel/ivsc/Makefile | 7 + drivers/media/pci/intel/ivsc/csi_bridge.c | 332 +++++++++++++ drivers/media/pci/intel/ivsc/csi_bridge.h | 122 +++++ drivers/media/pci/intel/ivsc/mei_csi.c | 775++++++++++++++++++++++++++++++
Quoted 6 lines
7 files changed, 1251 insertions(+) create mode 100644 drivers/media/pci/intel/ivsc/Kconfig create mode 100644 drivers/media/pci/intel/ivsc/Makefile create mode 100644 drivers/media/pci/intel/ivsc/csi_bridge.c create mode 100644 drivers/media/pci/intel/ivsc/csi_bridge.h create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.cI'm commenting just
mei_csithis time...Quoted 199 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +192
-0@@ -0,0 +1,775 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright (C) 2023 Intel Corporation. All rights reserved. +* Intel Visual Sensing Controller CSI Linux driver */ + +/* +* To set ownership of CSI-2 link and to configure CSI-2 link, there +* are specific commands, which are sent via MEI protocol. The send +* command function uses "completion" as a synchronization mechanism. +* The response for command is received via a mei callback which +wakes +* up the caller. There can be only one outstanding command at a time. +*/ + +#include <linux/completion.h> +#include <linux/delay.h> +#include <linux/kernel.h> +#include <linux/math64.h> +#include <linux/mei_cl_bus.h> +#include <linux/module.h> +#include <linux/mutex.h> +#include <linux/pm_runtime.h> +#include <linux/slab.h> +#include <linux/units.h> +#include <linux/uuid.h> +#include <linux/workqueue.h> + +#include <media/v4l2-async.h> +#include <media/v4l2-ctrls.h> +#include <media/v4l2-fwnode.h> +#include <media/v4l2-subdev.h> + +#define MEI_CSI_DRIVER_NAME "ivsc_csi" + +/* the 5s used here is based on experiment */ #define CSI_CMD_TIMEOUT +(5 *HZ ) +/* to setup CSI-2 link an extra delay needed and determined +experimentally */ #define CSI_FW_READY_DELAY_MS 100 +/* link frequency unit is 100kHz */ +#define CSI_LINK_FREQ (x) ((u32)(div_u64(x, 100 * HZ_PER_KHZ))) + +/* +* identify the command id supported by firmware +* IPC, as well as the privacy notification id +* used when processing privacy event. +*/ +enum csi_cmd_id { +/* used to set csi ownership */ +CSI_SET_OWNER =0 , + +/* used to configure CSI-2 link */ +CSI_SET_CONF =2 , + +/* privacy notification id used when privacy state changes */ +CSI_PRIVACY_NOTIF =6 , +}; + +/* CSI-2 link ownership definition */ enum csi_link_owner { +CSI_LINK_IVSC , +CSI_LINK_HOST , +}; + +/* privacy status definition */ +enum ivsc_privacy_status { +CSI_PRIVACY_OFF , +CSI_PRIVACY_ON , +CSI_PRIVACY_MAX , +}; + +enum csi_pads { +CSI_PAD_SOURCE , +CSI_PAD_SINK , +CSI_NUM_PADS +}; + +/* configuration of the CSI-2 link between host and IVSC */ struct +csi_link_cfg { +/* number of data lanes used on the CSI-2 link */ +u32 nr_of_lanes; + +/* frequency of the CSI-2 link */ +u32 link_freq; + +/* for future use */ +u32 rsvd[2 ]; +} __packed; + +/* CSI command structure */ +struct csi_cmd { +u32 cmd_id; +union _cmd_param { +u32 param; +struct csi_link_cfg conf; + } param; +} __packed; + +/* CSI notification structure */ +struct csi_notif { +u32 cmd_id; +int status; +union _resp_cont { +u32 cont; +struct csi_link_cfg conf; + } cont; +} __packed; + +struct mei_csi { +struct mei_cl_device *cldev; + +/* command response */ +struct csi_notif cmd_response; +/* used to wait for command response from firmware */ +struct completion cmd_completion; +/* protect command download */ +struct mutex lock; + +struct v4l2_subdev subdev; +struct v4l2_subdev *remote; +struct v4l2_async_notifier notifier; +struct v4l2_ctrl_handler ctrl_handler; +struct v4l2_ctrl *privacy_ctrl; +unsigned int remote_pad; +/* start streaming or not */ +int streaming; + +struct media_pad pads[CSI_NUM_PADS ]; +struct v4l2_mbus_framefmt format_mbus[CSI_NUM_PADS ]; + +/* number of data lanes used on the CSI-2 link */ +u32 nr_of_lanes; +/* frequency of the CSI-2 link */ +u64 link_freq; + +/* privacy status */ +enum ivsc_privacy_status status; +}; + +static const struct v4l2_mbus_framefmt mei_csi_format_mbus_default = { + .width =1 , + .height =1 , + .code =MEDIA_BUS_FMT_Y8_1X8 , + .field =V4L2_FIELD_NONE , +}; + +int csi_bridge_init (struct mei_cl_device *csi_dev); + +static inline struct mei_csi *notifier_to_csi (struct +v4l2_async_notifier *n) { + return container_of(n, struct mei_csi, notifier); } + +static inline struct mei_csi *sd_to_csi (struct v4l2_subdev *sd) { + return container_of(sd, struct mei_csi, subdev); } + +/* send a command to firmware and mutex must be held by caller */ +static int mei_csi_send (struct mei_csi *csi,u8 *buf,size_t len) { +struct csi_cmd *cmd = (struct csi_cmd *)buf; +int ret; + +reinit_completion (&csi->cmd_completion); + + ret =mei_cldev_send (csi->cldev, buf, len); +if (ret <0 ) + goto out; + + ret =wait_for_completion_killable_timeout (&csi->cmd_completion, +CSI_CMD_TIMEOUT ); +if (ret <0 ) { + goto out; + }else if (!ret) { + ret = -ETIMEDOUT ; + goto out; + } + +/* command response status */ + ret = csi->cmd_response.status; +if (ret) { + ret = -EINVAL ; + goto out; + } + +if (csi->cmd_response.cmd_id != cmd->cmd_id) + ret = -EINVAL ; + +out: +return ret; +} + +/* set CSI-2 link ownership */ +static int csi_set_link_owner (struct mei_csi *csi,enum +csi_link_owner owner) { +struct csi_cmd cmd = {0 }; +size_t cmd_size;Quoted 8 lines
drivers/media/pci/intel/ivsc/mei_csi.c line 202
Quoted 5 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +5
-0+int ret; + + cmd.cmd_id =CSI_SET_OWNER ; + cmd.param.param = owner; + cmd_size =sizeof (cmd.cmd_id) +sizeof (cmd.param.param);In some cases you're using memset and in others not. If you don't need memset, I'd prefer assigning the fields in variable declaration instead.
Quoted 1 line
The declaration will be like below, but it will break reverse x-mas tree style.
Have I asked to change your e-mail client settings?
Quoted 4 lines
struct
csi_cmdcmd = { 0 };size_tcmd_size=sizeof(cmd.cmd_id)+ sizeof(cmd.param.param); int ret;
It's not a problem if you have a dependency.
Quoted 91 lines
Quoted 82 lines
Quoted 13 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +10
-0+ +mutex_lock (&csi->lock); + + ret =mei_csi_send (csi, (u8 *)&cmd, cmd_size); + +mutex_unlock (&csi->lock); + +return ret; +} +drivers/media/pci/intel/ivsc/mei_csi.c line 223
Quoted 10 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +10
-0+/* configure CSI-2 link between host and IVSC */ static int +csi_set_link_cfg (struct mei_csi *csi) { +struct csi_cmd cmd = {0 }; +size_t cmd_size; +int ret; + + cmd.cmd_id =CSI_SET_CONF ; + cmd.param.conf.nr_of_lanes = csi->nr_of_lanes; + cmd.param.conf.link_freq =CSI_LINK_FREQ (csi->link_freq); + cmd_size =sizeof (cmd.cmd_id) +sizeof (cmd.param.conf);Ditto.
Quoted 56 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +56
-0+ +mutex_lock (&csi->lock); + + ret =mei_csi_send (csi, (u8 *)&cmd, cmd_size); +/* +* wait configuration ready if download success. placing +* delay under mutex is to make sure current command flow +* completed before starting a possible new one. +*/ +if (!ret) +msleep (CSI_FW_READY_DELAY_MS ); + +mutex_unlock (&csi->lock); + +return ret; +} + +/* callback for receive */ +static void mei_csi_rx (struct mei_cl_device *cldev) { +struct mei_csi *csi =mei_cldev_get_drvdata (cldev); +struct csi_notif notif = {0 }; +int ret; + + ret =mei_cldev_recv (cldev, (u8 *)¬if,sizeof (notif)); +if (ret <0 ) { +dev_err (&cldev->dev,"recv error: %d\n" , ret); +return ; + } + +switch (notif.cmd_id) { +case CSI_PRIVACY_NOTIF : +if (notif.cont.cont <CSI_PRIVACY_MAX ) { + csi->status = notif.cont.cont; +v4l2_ctrl_s_ctrl (csi->privacy_ctrl, csi->status); + } +break ; +case CSI_SET_OWNER : +case CSI_SET_CONF : +memcpy (&csi->cmd_response, ¬if, ret); + +complete (&csi->cmd_completion); +break ; +default : +break ; + } +} + +static int mei_csi_pre_streamon (struct v4l2_subdev *sd,u32 flags) { +struct v4l2_querymenu qm = { .id =V4L2_CID_LINK_FREQ , }; +struct v4l2_mbus_config mbus_config = {0 }; +struct mei_csi *csi =sd_to_csi (sd); +struct v4l2_ctrl *ctrl; +int ret =0 ; + +if (!csi->remote) +return -ENODEV ;Quoted 8 lines
drivers/media/pci/intel/ivsc/mei_csi.c line 286
Quoted 5 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +5
-0+ + ret =v4l2_subdev_call (csi->remote, pad, get_mbus_config, + csi->remote_pad, &mbus_config); +if (ret) +return ret;You're already parsing the endpoint in
probe(). Do you need this, assuming the sensor has a static lane configuration? Virtually all do.
Quoted 2 lines
When parsing remote endpoint, just get
num_data_lanesbyv4l2_fwnode_endpoint_alloc_parse?
Yes.
Quoted 5 lines
Quoted 3 lines
Could this all be done via the
s_stream()op instead?We don't need csi2 link freq and lane number when
s_steam(0)
You can set the VOLATILE flag on the control and then query the upstream sub-device in the g_ctrl callback.
Quoted 117 lines
Quoted 42 lines
Quoted 42 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +40
-0+ +if (mbus_config.type !=V4L2_MBUS_CSI2_DPHY ) +return -EINVAL ; + + csi->nr_of_lanes = mbus_config.bus.mipi_csi2.num_data_lanes; + + ctrl =v4l2_ctrl_find (csi->remote->ctrl_handler,V4L2_CID_LINK_FREQ ); +if (!ctrl) +return -EINVAL ; + qm.index =v4l2_ctrl_g_ctrl (ctrl); + + ret =v4l2_querymenu (csi->remote->ctrl_handler, &qm); +if (ret) +return ret; + + csi->link_freq = qm.value; + +return ret; +} + +static int mei_csi_set_stream (struct v4l2_subdev *sd,int enable) { +struct mei_csi *csi =sd_to_csi (sd); +int ret =0 ; + +if (enable && csi->streaming ==0 ) { +/* switch CSI-2 link to host */ + ret =csi_set_link_owner (csi,CSI_LINK_HOST ); +if (ret <0 ) + goto err; + +/* configure CSI-2 link */ + ret =csi_set_link_cfg (csi); +if (ret <0 ) + goto err; + + ret =v4l2_subdev_call (csi->remote, video, s_stream,1 ); +if (ret) + goto err; + }else if (!enable && csi->streaming ==1 ) { +v4l2_subdev_call (csi->remote, video, s_stream,0 );Quoted 8 lines
drivers/media/pci/intel/ivsc/mei_csi.c line 332
Quoted 5 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +5
-0+ +/* switch CSI-2 link to IVSC */ + ret =csi_set_link_owner (csi,CSI_LINK_IVSC ); +if (ret <0 ) + goto err;I'd complain (e.g.
dev_warn()) but return zero in this case. There's not much if anything the caller can do with this.ack
Quoted 56 lines
Quoted 56 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +55
-0+ } + + csi->streaming = enable; + +err: +return ret; +} + +static struct v4l2_mbus_framefmt * +mei_csi_get_pad_format (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +unsigned int pad,u32 which) { +struct mei_csi *csi =sd_to_csi (sd); + +switch (which) { +case V4L2_SUBDEV_FORMAT_TRY : +return v4l2_subdev_get_try_format (sd, sd_state, pad); +case V4L2_SUBDEV_FORMAT_ACTIVE : +return &csi->format_mbus[pad]; +default : +return NULL ; + } +} + +static int mei_csi_init_cfg (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state) { +struct v4l2_mbus_framefmt *mbusformat; +struct mei_csi *csi =sd_to_csi (sd); +unsigned int i; + +mutex_lock (&csi->lock); + +for (i =0 ; i < sd->entity.num_pads; i++) { + mbusformat =v4l2_subdev_get_try_format (sd, sd_state, i); + *mbusformat = mei_csi_format_mbus_default; + } + +mutex_unlock (&csi->lock); + +return 0 ; +} + +static int mei_csi_get_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *format) { +struct v4l2_mbus_framefmt *mbusformat; +struct mei_csi *csi =sd_to_csi (sd); + +mutex_lock (&csi->lock); + + mbusformat =mei_csi_get_pad_format (sd, sd_state, format->pad, + format->which); +if (mbusformat) + format->format = *mbusformat; +Quoted 8 lines
drivers/media/pci/intel/ivsc/mei_csi.c line 375
Quoted 4 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +4
-0+mutex_unlock (&csi->lock); + +return 0 ; +}It'd be nice to have also
enum_mbus_codesupport. Video mux of course didn't have this either. Something that could be done in the framework, including validating the mbus format.
Quoted 1 line
Sorry, you mean the default
MEDIA_BUS_FMT_Y8_1X8?
Any format that can be supported by the device.
The video mux should have that, too. I'll see if this would fit to the framework neatly somehow.
Quoted 194 lines
Quoted 194 lines
Quoted 194 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +162
-0+ +static int mei_csi_set_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *format) { +struct v4l2_mbus_framefmt *source_mbusformat; +struct v4l2_mbus_framefmt *mbusformat; +struct mei_csi *csi =sd_to_csi (sd); +struct media_pad *pad; + + mbusformat =mei_csi_get_pad_format (sd, sd_state, format->pad, + format->which); +if (!mbusformat) +return -EINVAL ; + + source_mbusformat =mei_csi_get_pad_format (sd, sd_state, +CSI_PAD_SOURCE , + format->which); +if (!source_mbusformat) +return -EINVAL ; + +v4l_bound_align_image (&format->format.width,1 ,65536 ,0 , + &format->format.height,1 ,65536 ,0 ,0 ); + +switch (format->format.code) { +case MEDIA_BUS_FMT_RGB444_1X12 : +case MEDIA_BUS_FMT_RGB444_2X8_PADHI_BE : +case MEDIA_BUS_FMT_RGB444_2X8_PADHI_LE : +case MEDIA_BUS_FMT_RGB555_2X8_PADHI_BE : +case MEDIA_BUS_FMT_RGB555_2X8_PADHI_LE : +case MEDIA_BUS_FMT_RGB565_1X16 : +case MEDIA_BUS_FMT_BGR565_2X8_BE : +case MEDIA_BUS_FMT_BGR565_2X8_LE : +case MEDIA_BUS_FMT_RGB565_2X8_BE : +case MEDIA_BUS_FMT_RGB565_2X8_LE : +case MEDIA_BUS_FMT_RGB666_1X18 : +case MEDIA_BUS_FMT_RBG888_1X24 : +case MEDIA_BUS_FMT_RGB666_1X24_CPADHI : +case MEDIA_BUS_FMT_BGR888_1X24 : +case MEDIA_BUS_FMT_GBR888_1X24 : +case MEDIA_BUS_FMT_RGB888_1X24 : +case MEDIA_BUS_FMT_RGB888_2X12_BE : +case MEDIA_BUS_FMT_RGB888_2X12_LE : +case MEDIA_BUS_FMT_ARGB8888_1X32 : +case MEDIA_BUS_FMT_RGB888_1X32_PADHI : +case MEDIA_BUS_FMT_RGB101010_1X30 : +case MEDIA_BUS_FMT_RGB121212_1X36 : +case MEDIA_BUS_FMT_RGB161616_1X48 : +case MEDIA_BUS_FMT_Y8_1X8 : +case MEDIA_BUS_FMT_UV8_1X8 : +case MEDIA_BUS_FMT_UYVY8_1_5X8 : +case MEDIA_BUS_FMT_VYUY8_1_5X8 : +case MEDIA_BUS_FMT_YUYV8_1_5X8 : +case MEDIA_BUS_FMT_YVYU8_1_5X8 : +case MEDIA_BUS_FMT_UYVY8_2X8 : +case MEDIA_BUS_FMT_VYUY8_2X8 : +case MEDIA_BUS_FMT_YUYV8_2X8 : +case MEDIA_BUS_FMT_YVYU8_2X8 : +case MEDIA_BUS_FMT_Y10_1X10 : +case MEDIA_BUS_FMT_UYVY10_2X10 : +case MEDIA_BUS_FMT_VYUY10_2X10 : +case MEDIA_BUS_FMT_YUYV10_2X10 : +case MEDIA_BUS_FMT_YVYU10_2X10 : +case MEDIA_BUS_FMT_Y12_1X12 : +case MEDIA_BUS_FMT_UYVY12_2X12 : +case MEDIA_BUS_FMT_VYUY12_2X12 : +case MEDIA_BUS_FMT_YUYV12_2X12 : +case MEDIA_BUS_FMT_YVYU12_2X12 : +case MEDIA_BUS_FMT_UYVY8_1X16 : +case MEDIA_BUS_FMT_VYUY8_1X16 : +case MEDIA_BUS_FMT_YUYV8_1X16 : +case MEDIA_BUS_FMT_YVYU8_1X16 : +case MEDIA_BUS_FMT_YDYUYDYV8_1X16 : +case MEDIA_BUS_FMT_UYVY10_1X20 : +case MEDIA_BUS_FMT_VYUY10_1X20 : +case MEDIA_BUS_FMT_YUYV10_1X20 : +case MEDIA_BUS_FMT_YVYU10_1X20 : +case MEDIA_BUS_FMT_VUY8_1X24 : +case MEDIA_BUS_FMT_YUV8_1X24 : +case MEDIA_BUS_FMT_UYYVYY8_0_5X24 : +case MEDIA_BUS_FMT_UYVY12_1X24 : +case MEDIA_BUS_FMT_VYUY12_1X24 : +case MEDIA_BUS_FMT_YUYV12_1X24 : +case MEDIA_BUS_FMT_YVYU12_1X24 : +case MEDIA_BUS_FMT_YUV10_1X30 : +case MEDIA_BUS_FMT_UYYVYY10_0_5X30 : +case MEDIA_BUS_FMT_AYUV8_1X32 : +case MEDIA_BUS_FMT_UYYVYY12_0_5X36 : +case MEDIA_BUS_FMT_YUV12_1X36 : +case MEDIA_BUS_FMT_YUV16_1X48 : +case MEDIA_BUS_FMT_UYYVYY16_0_5X48 : +case MEDIA_BUS_FMT_JPEG_1X8 : +case MEDIA_BUS_FMT_AHSV8888_1X32 : +case MEDIA_BUS_FMT_SBGGR8_1X8 : +case MEDIA_BUS_FMT_SGBRG8_1X8 : +case MEDIA_BUS_FMT_SGRBG8_1X8 : +case MEDIA_BUS_FMT_SRGGB8_1X8 : +case MEDIA_BUS_FMT_SBGGR10_1X10 : +case MEDIA_BUS_FMT_SGBRG10_1X10 : +case MEDIA_BUS_FMT_SGRBG10_1X10 : +case MEDIA_BUS_FMT_SRGGB10_1X10 : +case MEDIA_BUS_FMT_SBGGR12_1X12 : +case MEDIA_BUS_FMT_SGBRG12_1X12 : +case MEDIA_BUS_FMT_SGRBG12_1X12 : +case MEDIA_BUS_FMT_SRGGB12_1X12 : +case MEDIA_BUS_FMT_SBGGR14_1X14 : +case MEDIA_BUS_FMT_SGBRG14_1X14 : +case MEDIA_BUS_FMT_SGRBG14_1X14 : +case MEDIA_BUS_FMT_SRGGB14_1X14 : +case MEDIA_BUS_FMT_SBGGR16_1X16 : +case MEDIA_BUS_FMT_SGBRG16_1X16 : +case MEDIA_BUS_FMT_SGRBG16_1X16 : +case MEDIA_BUS_FMT_SRGGB16_1X16 : +break ; +default : + format->format.code =MEDIA_BUS_FMT_Y8_1X8 ; +break ; + } +if (format->format.field ==V4L2_FIELD_ANY ) + format->format.field =V4L2_FIELD_NONE ; + +mutex_lock (&csi->lock); + + pad = &csi->pads[format->pad]; +if (pad->flags &MEDIA_PAD_FL_SOURCE ) + format->format = csi->format_mbus[CSI_PAD_SINK ]; + + *mbusformat = format->format; + +if (pad->flags &MEDIA_PAD_FL_SINK ) + *source_mbusformat = format->format; + +mutex_unlock (&csi->lock); + +return 0 ; +} + +static const struct v4l2_subdev_video_ops mei_csi_video_ops = { + .s_stream = mei_csi_set_stream, + .pre_streamon = mei_csi_pre_streamon, }; + +static const struct v4l2_subdev_pad_ops mei_csi_pad_ops = { + .init_cfg = mei_csi_init_cfg, + .get_fmt = mei_csi_get_fmt, + .set_fmt = mei_csi_set_fmt, +}; + +static const struct v4l2_subdev_ops mei_csi_subdev_ops = { + .video = &mei_csi_video_ops, + .pad = &mei_csi_pad_ops, +}; + +static const struct media_entity_operations mei_csi_entity_ops = { + .link_validate = v4l2_subdev_link_validate, }; + +static int mei_csi_notify_bound (struct v4l2_async_notifier *notifier, +struct v4l2_subdev *subdev, +struct v4l2_async_subdev *asd) +{ +struct mei_csi *csi =notifier_to_csi (notifier); +int pad; + + pad = media_entity_get_fwnode_pad(&subdev->entity, asd-match.fwnode,
drivers/media/pci/intel/ivsc/mei_csi.c new file +29
-0+ MEDIA_PAD_FL_SOURCE); +if (pad <0 ) +return pad; + + csi->remote = subdev; + csi->remote_pad = pad; + +return media_create_pad_link (&subdev->entity, pad, + &csi->subdev.entity,0 , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); +} + +static void mei_csi_notify_unbind (struct v4l2_async_notifier *notifier, +struct v4l2_subdev *subdev, +struct v4l2_async_subdev *asd) +{ +struct mei_csi *csi =notifier_to_csi (notifier); + + csi->remote =NULL ; +} + +static const struct v4l2_async_notifier_operations mei_csi_notify_ops = { + .bound = mei_csi_notify_bound, + .unbind = mei_csi_notify_unbind, +}; + +static int mei_csi_init_control (struct mei_csi *csi) { +v4l2_ctrl_handler_init (&csi->ctrl_handler,1 );
Quoted 8 lines
Quoted 8 lines
drivers/media/pci/intel/ivsc/mei_csi.c line 595
Quoted 6 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +6
-0+ csi->ctrl_handler.lock = &csi->lock; + + csi->privacy_ctrl =v4l2_ctrl_new_std (&csi->ctrl_handler,NULL , +V4L2_CID_PRIVACY ,0 ,1 ,1 ,0 ); +if (csi->ctrl_handler.error) +return csi->ctrl_handler.error;The control should be made volatile here (flags |=
Read-only, I meant. Volatile is for controls that are not driven by hardware events somehow.
Quoted 199 lines
Quoted 1 line
V4L2_CTRL_FLAG_READ_ONLY) as it can't be changed by the user.ack
Quoted 93 lines
Quoted 93 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +92
-0+ + csi->subdev.ctrl_handler = &csi->ctrl_handler; + +return 0 ; +} + +static int mei_csi_parse_firmware (struct mei_csi *csi) { +struct v4l2_fwnode_endpoint v4l2_ep = { + .bus_type =V4L2_MBUS_CSI2_DPHY , + }; +struct device *dev = &csi->cldev->dev; +struct v4l2_async_subdev *asd; +struct fwnode_handle *fwnode; +struct fwnode_handle *ep; +int ret; + + ep =fwnode_graph_get_endpoint_by_id (dev_fwnode (dev),0 ,0 ,0 ); +if (!ep) { +dev_err (dev,"not connected to subdevice\n" ); +return -EINVAL ; + } + + ret =v4l2_fwnode_endpoint_parse (ep, &v4l2_ep); +if (ret) { +dev_err (dev,"could not parse v4l2 endpoint\n" ); +fwnode_handle_put (ep); +return -EINVAL ; + } + + fwnode =fwnode_graph_get_remote_endpoint (ep); +fwnode_handle_put (ep); + +v4l2_async_nf_init (&csi->notifier); + csi->notifier.ops = &mei_csi_notify_ops; + + asd =v4l2_async_nf_add_fwnode (&csi->notifier, fwnode, + struct v4l2_async_subdev); +fwnode_handle_put (fwnode); +if (IS_ERR (asd)) +return PTR_ERR (asd); + + ret =v4l2_async_subdev_nf_register (&csi->subdev, &csi->notifier); +if (ret) +v4l2_async_nf_cleanup (&csi->notifier); + +return ret; +} + +static int mei_csi_probe (struct mei_cl_device *cldev, +const struct mei_cl_device_id *id) { +struct mei_csi *csi; +int ret; + +/* +* no connections to sensor are defined in firmware, try to +* build connections as software_nodes parsed from SSDB. +*/ + ret =csi_bridge_init (cldev); +if (ret) +return ret; + + csi =devm_kzalloc (&cldev->dev,sizeof (struct mei_csi ),GFP_KERNEL ); +if (!csi) +return -ENOMEM ; + + csi->cldev = cldev; +mutex_init (&csi->lock); +init_completion (&csi->cmd_completion); + +mei_cldev_set_drvdata (cldev, csi); + + ret =mei_cldev_enable (cldev); +if (ret <0 ) { +dev_err (&cldev->dev,"mei_cldev_enable failed: %d\n" , ret); + goto destroy_mutex; + } + + ret =mei_cldev_register_rx_cb (cldev, mei_csi_rx); +if (ret) { +dev_err (&cldev->dev,"event cb registration failed: %d\n" , ret); + goto err_disable; + } + + ret =mei_csi_parse_firmware (csi); +if (ret) + goto err_disable; + + csi->subdev.dev = &cldev->dev; +v4l2_subdev_init (&csi->subdev, &mei_csi_subdev_ops); +v4l2_set_subdevdata (&csi->subdev, csi); + csi->subdev.flags =V4L2_SUBDEV_FL_HAS_DEVNODE ; + csi->subdev.entity.function =MEDIA_ENT_F_VID_IF_BRIDGE ;Quoted 8 lines
drivers/media/pci/intel/ivsc/mei_csi.c line 694
Quoted 5 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +5
-0+ csi->subdev.entity.ops = &mei_csi_entity_ops; + + ret =mei_csi_init_control (csi); +if (ret) + goto err_async;You'll also need to call
v4l2_ctrl_handler_free()if setting up the handler failed. I.e. just change the label here and removeerr_asynclabel below.Thanks
Quoted 29 lines
Quoted 29 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +28
-0+ + csi->format_mbus[CSI_PAD_SOURCE ] = mei_csi_format_mbus_default; + csi->format_mbus[CSI_PAD_SINK ] = mei_csi_format_mbus_default; + + csi->pads[CSI_PAD_SOURCE ].flags =MEDIA_PAD_FL_SOURCE ; + csi->pads[CSI_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; + ret =media_entity_pads_init (&csi->subdev.entity,CSI_NUM_PADS , + csi->pads); +if (ret) + goto err_ctrl_handler; + + ret =v4l2_subdev_init_finalize (&csi->subdev); +if (ret <0 ) + goto err_entity; + + ret =v4l2_async_register_subdev (&csi->subdev); +if (ret <0 ) + goto err_subdev; + +pm_runtime_enable (&cldev->dev); + +return 0 ; + +err_subdev: +v4l2_subdev_cleanup (&csi->subdev); +err_entity: +media_entity_cleanup (&csi->subdev.entity); +err_ctrl_handler:Quoted 8 lines
drivers/media/pci/intel/ivsc/mei_csi.c line 728
Quoted 6 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +6
-0+v4l2_ctrl_handler_free (&csi->ctrl_handler); +err_async: +v4l2_async_nf_unregister (&csi->notifier); +v4l2_async_nf_cleanup (&csi->notifier); +err_disable: +mei_cldev_disable (cldev);It'd be nice to have an additional newline before the label. Elsewhere, too.
ack
Quoted 31 lines
Quoted 18 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +17
-0+destroy_mutex: +mutex_destroy (&csi->lock); + +return ret; +} + +static void mei_csi_remove (struct mei_cl_device *cldev) { +struct mei_csi *csi =mei_cldev_get_drvdata (cldev); + +v4l2_async_nf_unregister (&csi->notifier); +v4l2_async_nf_cleanup (&csi->notifier); +v4l2_ctrl_handler_free (&csi->ctrl_handler); +v4l2_async_unregister_subdev (&csi->subdev); +v4l2_subdev_cleanup (&csi->subdev); +media_entity_cleanup (&csi->subdev.entity); + +pm_runtime_disable (&cldev->dev);drivers/media/pci/intel/ivsc/mei_csi.c line 756
Quoted 10 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +10
-0+ +mutex_destroy (&csi->lock); +} + +#define MEI_CSI_UUID UUID_LE(0x92335FCF, 0x3203, 0x4472, \ + 0xAF, 0x93, 0x7b, 0x44, 0x53, 0xAC, 0x29, 0xDA) + +static const struct mei_cl_device_id mei_csi_tbl[] = { + {MEI_CSI_DRIVER_NAME ,MEI_CSI_UUID ,MEI_CL_VERSION_ANY }, +Extra newline.
Quoted 4 lines
drivers/media/pci/intel/ivsc/mei_csi.c line 757
Quoted 1 line
drivers/media/pci/intel/ivsc/mei_csi.c new file +1
-0+/* required last entry */ The comment could be on the same line below. E.g. "Sentinel" is appropriate, as this isn't just any last entry.
thanks
Quoted 20 lines
Quoted 19 lines
drivers/media/pci/intel/ivsc/mei_csi.c new file +18
-0+ { } +}; +MODULE_DEVICE_TABLE (mei, mei_csi_tbl); + +static struct mei_cl_driver mei_csi_driver = { + .id_table = mei_csi_tbl, + .name =MEI_CSI_DRIVER_NAME , + + .probe = mei_csi_probe, + .remove = mei_csi_remove, +}; + +module_mei_cl_driver (mei_csi_driver); + +MODULE_AUTHOR ("Wentong Wu <[email protected]>" ); +MODULE_AUTHOR ("Zhifeng Wang <[email protected]>" ); +MODULE_DESCRIPTION ("Device driver for IVSC CSI" ); +MODULE_LICENSE ("GPL" );
Signature · 3 lines
Kind regards, Sakari Ailus
Sakari Ailus
maintainerRe: [PATCH v3 1/3] media: pci: intel: ivsc: Add CSI submodule
In reply to Wentong WuHi Wentong,
drivers/media/pci/intel/ivsc/Kconfig line 6
Quoted 12 lines · Wentong Wu
drivers/media/pci/intel/ivsc/Kconfig new file +6
-0@@ -0,0 +1,12 @@ +# SPDX-License-Identifier: GPL-2.0-only +# Copyright (C) 2023, Intel Corporation. All rights reserved. + +config INTEL_VSC + tristate "Intel Visual Sensing Controller" + depends on INTEL_MEI
I think you should depend on PM here.
Alternatively, request host ownership from ace + csi in probe() if runtime PM isn't enabled.
Quoted 6 lines
drivers/media/pci/intel/ivsc/Kconfig new file +6
-0+ help + This adds support for Intel Visual Sensing Controller (IVSC). + + Enables the IVSC firmware services required for controlling + camera sensor ownership and CSI-2 link through Image Processing + Unit(IPU) driver of Intel.
Signature · 3 lines
Kind regards, Sakari Ailus
PATCH v3 2/3 media: pci: intel: ivsc: Add ACE submodule
In reply to Wentong WuACE is a submodule of IVSC which controls camera sensor's ownership, belonging to host or IVSC. When IVSC owns camera sensor, it is for algorithm computing. When host wants to control camera sensor, ACE module needs to be informed of ownership with defined interface.
The interface is via MEI. There is a separate MEI UUID, which this driver uses to enumerate.
To switch ownership of camera sensor between IVSC and host, the caller specifies the defined ownership information which will be sent to firmware by sending MEI command.
Device link(device_link_add) is used to set the right camera sensor ownership before accessing the sensor via I²C. With DL_FLAG_PM_RUNTIME and DL_FLAG_RPM_ACTIVE, the supplier device will be PM runtime resumed before the consumer(camera sensor). So use runtime PM callbacks to transfer the ownership between host and IVSC.
--- drivers/media/pci/intel/ivsc/Makefile | 3 + drivers/media/pci/intel/ivsc/mei_ace.c | 534 +++++++++++++++++++++++++++++++++ 2 files changed, 537 insertions(+) create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
drivers/media/pci/intel/ivsc/Makefile +3 -0
@@ -5,3 +5,6 @@ obj-$(CONFIG_INTEL_VSC) += ivsc-csi.o ivsc-csi-y += mei_csi.o ivsc-csi-y += csi_bridge.o + +obj-$(CONFIG_INTEL_VSC) += ivsc-ace.o +ivsc-ace-y += mei_ace.o
drivers/media/pci/intel/ivsc/mei_ace.c new file +534 -0
534 lines not on this page · raw message
Signature · 1 line
2.7.4
Sakari Ailus
maintainerRe: [PATCH v3 2/3] media: pci: intel: ivsc: Add ACE submodule
Hi Wentong,
Quoted 48 lines · Wentong Wu
ACE is a submodule of IVSC which controls camera sensor's ownership, belonging to host or IVSC. When IVSC owns camera sensor, it is for algorithm computing. When host wants to control camera sensor, ACE module needs to be informed of ownership with defined interface.
The interface is via MEI. There is a separate MEI UUID, which this driver uses to enumerate.
To switch ownership of camera sensor between IVSC and host, the caller specifies the defined ownership information which will be sent to firmware by sending MEI command.
Device
link(device_link_add)is used to set the right camera sensor ownership before accessing the sensor via I²C. WithDL_FLAG_PM_RUNTIMEandDL_FLAG_RPM_ACTIVE, the supplier device will be PM runtime resumed before the consumer(camera sensor). So use runtime PM callbacks to transfer the ownership between host and IVSC.Signed-off-by Wentong Wu <[email protected]>--- drivers/media/pci/intel/ivsc/Makefile | 3 + drivers/media/pci/intel/ivsc/mei_ace.c | 534 +++++++++++++++++++++++++++++++++ 2 files changed, 537 insertions(+) create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.cdrivers/media/pci/intel/ivsc/Makefile +3
-0@@ -5,3 +5,6 @@ obj-$(CONFIG_INTEL_VSC) += ivsc-csi.o ivsc-csi-y += mei_csi.o ivsc-csi-y += csi_bridge.o + +obj-$(CONFIG_INTEL_VSC) += ivsc-ace.o +ivsc-ace-y += mei_ace.odrivers/media/pci/intel/ivsc/mei_ace.c new file +4
-0@@ -0,0 +1,534 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright (C) 2023 Intel Corporation. All rights reserved. +* Intel Visual Sensing Controller ACE Linux driver
drivers/media/pci/intel/ivsc/mei_ace.c line 14
Quoted 10 lines
drivers/media/pci/intel/ivsc/mei_ace.c new file +10
-0+*/ + +/* +* To set ownership of camera sensor, there is specific command, which +* is sent via MEI protocol. That's a two-step scheme where the firmware +* first acks receipt of the command and later responses the command was +* executed. The command sending function uses "completion" as the +* synchronization mechanism. The notification for command is received +* via a mei callback which wakes up the caller. There can be only one +* outstanding command at a time.
Could you document the dependencies in the sensor initialisation (ace + csi) and how the sensor is powered (i.e. no direct control of the PMIC from the host?)? This doesn't seem like a bad place to do it.
Quoted 264 lines
drivers/media/pci/intel/ivsc/mei_ace.c new file +264
-0+*/ + +#include <linux/acpi.h> +#include <linux/completion.h> +#include <linux/delay.h> +#include <linux/kernel.h> +#include <linux/mei_cl_bus.h> +#include <linux/module.h> +#include <linux/mutex.h> +#include <linux/pm_runtime.h> +#include <linux/slab.h> +#include <linux/uuid.h> + +#define MEI_ACE_DRIVER_NAME "ivsc_ace" + +/* indicating driver message */ +#define ACE_DRV_MSG 1 +/* indicating set command */ +#define ACE_CMD_SET 4 +/* command timeout determined experimentally */ +#define ACE_CMD_TIMEOUT (5 * HZ) +/* indicating the first command block */ +#define ACE_CMD_INIT_BLOCK 1 +/* indicating the last command block */ +#define ACE_CMD_FINAL_BLOCK 1 +/* size of camera status notification content */ +#define ACE_CAMERA_STATUS_SIZE 5 + +/* UUID used to get firmware id */ +#define ACE_GET_FW_ID_UUID UUID_LE(0x6167DCFB, 0x72F1, 0x4584, 0xBF, \ + 0xE3, 0x84, 0x17, 0x71, 0xAA, 0x79, 0x0B) + +/* UUID used to get csi device */ +#define MEI_CSI_UUID UUID_LE(0x92335FCF, 0x3203, 0x4472, \ + 0xAF, 0x93, 0x7b, 0x44, 0x53, 0xAC, 0x29, 0xDA) + +/* identify firmware event type */ +enum ace_event_type { +/* firmware ready */ +ACE_FW_READY =0x8 , + +/* command response */ +ACE_CMD_RESPONSE =0x10 , +}; + +/* identify camera sensor ownership */ +enum ace_camera_owner { +ACE_CAMERA_IVSC , +ACE_CAMERA_HOST , +}; + +/* identify the command id supported by firmware IPC */ +enum ace_cmd_id { +/* used to switch camera sensor to host */ +ACE_SWITCH_CAMERA_TO_HOST =0x13 , + +/* used to switch camera sensor to IVSC */ +ACE_SWITCH_CAMERA_TO_IVSC =0x14 , + +/* used to get firmware id */ +ACE_GET_FW_ID =0x1A , +}; + +/* ACE command header structure */ +struct ace_cmd_hdr { +u32 firmware_id :16 ; +u32 instance_id :8 ; +u32 type :5 ; +u32 rsp :1 ; +u32 msg_tgt :1 ; +u32 _hw_rsvd_1 :1 ; +u32 param_size :20 ; +u32 cmd_id :8 ; +u32 final_block :1 ; +u32 init_block :1 ; +u32 _hw_rsvd_2 :2 ; +} __packed; + +/* ACE command parameter structure */ +union ace_cmd_param { +uuid_le uuid; +u32 param; +}; + +/* ACE command structure */ +struct ace_cmd { +struct ace_cmd_hdr hdr; +union ace_cmd_param param; +} __packed; + +/* ACE notification header */ +union ace_notif_hdr { +struct _confirm { +u32 status :24 ; +u32 type :5 ; +u32 rsp :1 ; +u32 msg_tgt :1 ; +u32 _hw_rsvd_1 :1 ; +u32 param_size :20 ; +u32 cmd_id :8 ; +u32 final_block :1 ; +u32 init_block :1 ; +u32 _hw_rsvd_2 :2 ; + } __packed ack; + +struct _event { +u32 rsvd1 :16 ; +u32 event_type :8 ; +u32 type :5 ; +u32 ack :1 ; +u32 msg_tgt :1 ; +u32 _hw_rsvd_1 :1 ; +u32 rsvd2 :30 ; +u32 _hw_rsvd_2 :2 ; + } __packed event; + +struct _response { +u32 event_id :16 ; +u32 notif_type :8 ; +u32 type :5 ; +u32 rsp :1 ; +u32 msg_tgt :1 ; +u32 _hw_rsvd_1 :1 ; +u32 event_data_size :16 ; +u32 request_target :1 ; +u32 request_type :5 ; +u32 cmd_id :8 ; +u32 _hw_rsvd_2 :2 ; + } __packed response; +}; + +/* ACE notification content */ +union ace_notif_cont { +u16 firmware_id; +u8 state_notif; +u8 camera_status[ACE_CAMERA_STATUS_SIZE ]; +}; + +/* ACE notification structure */ +struct ace_notif { +union ace_notif_hdr hdr; +union ace_notif_cont cont; +} __packed; + +struct mei_ace { +struct mei_cl_device *cldev; + +/* command ack */ +struct ace_notif cmd_ack; +/* command response */ +struct ace_notif cmd_response; +/* used to wait for command ack and response */ +struct completion cmd_completion; +/* lock used to prevent multiple call to ace */ +struct mutex lock; + +/* used to construct command */ +u16 firmware_id; + +/* runtime PM link from ace to csi */ +struct device_link *csi_link; +/* runtime PM link from ace to sensor */ +struct device_link *sensor_link; +}; + +static inline void init_cmd_hdr (struct ace_cmd_hdr *hdr) +{ +memset (hdr,0 ,sizeof (struct ace_cmd_hdr )); + + hdr->type =ACE_CMD_SET ; + hdr->msg_tgt =ACE_DRV_MSG ; + hdr->init_block =ACE_CMD_INIT_BLOCK ; + hdr->final_block =ACE_CMD_FINAL_BLOCK ; +} + +static int construct_command (struct mei_ace *ace,struct ace_cmd *cmd, +enum ace_cmd_id cmd_id) +{ +union ace_cmd_param *param = &cmd->param; +struct ace_cmd_hdr *hdr = &cmd->hdr; + +init_cmd_hdr (hdr); + + hdr->cmd_id = cmd_id; +switch (cmd_id) { +case ACE_GET_FW_ID : + param->uuid =ACE_GET_FW_ID_UUID ; + hdr->param_size =sizeof (param->uuid); +break ; +case ACE_SWITCH_CAMERA_TO_IVSC : + param->param =0 ; + hdr->firmware_id = ace->firmware_id; + hdr->param_size =sizeof (param->param); +break ; +case ACE_SWITCH_CAMERA_TO_HOST : + hdr->firmware_id = ace->firmware_id; +break ; +default : +return -EINVAL ; + } + +return hdr->param_size +sizeof (cmd->hdr); +} + +/* send a command to firmware and mutex must be held by caller */ +static int mei_ace_send (struct mei_ace *ace,struct ace_cmd *cmd, +size_t len,bool only_ack) +{ +union ace_notif_hdr *resp_hdr = &ace->cmd_response.hdr; +union ace_notif_hdr *ack_hdr = &ace->cmd_ack.hdr; +struct ace_cmd_hdr *cmd_hdr = &cmd->hdr; +int ret; + +reinit_completion (&ace->cmd_completion); + + ret =mei_cldev_send (ace->cldev, (u8 *)cmd, len); +if (ret <0 ) + goto out; + + ret =wait_for_completion_killable_timeout (&ace->cmd_completion, +ACE_CMD_TIMEOUT ); +if (ret <0 ) { + goto out; + }else if (!ret) { + ret = -ETIMEDOUT ; + goto out; + } + +if (ack_hdr->ack.cmd_id != cmd_hdr->cmd_id) { + ret = -EINVAL ; + goto out; + } + +/* command ack status */ + ret = ack_hdr->ack.status; +if (ret) { + ret = -EIO ; + goto out; + } + +if (only_ack) + goto out; + + ret =wait_for_completion_killable_timeout (&ace->cmd_completion, +ACE_CMD_TIMEOUT ); +if (ret <0 ) { + goto out; + }else if (!ret) { + ret = -ETIMEDOUT ; + goto out; + } + +if (resp_hdr->response.cmd_id != cmd_hdr->cmd_id) + ret = -EINVAL ; + +out: +return ret; +} + +static int ace_set_camera_owner (struct mei_ace *ace,enum ace_camera_owner owner) +{ +enum ace_cmd_id cmd_id; +struct ace_cmd cmd; +int cmd_size;
drivers/media/pci/intel/ivsc/mei_ace.c line 288
Quoted 10 lines
drivers/media/pci/intel/ivsc/mei_ace.c new file +10
-0+int ret; + +if (owner ==ACE_CAMERA_IVSC ) + cmd_id =ACE_SWITCH_CAMERA_TO_IVSC ; +else + cmd_id =ACE_SWITCH_CAMERA_TO_HOST ; + +mutex_lock (&ace->lock); + + cmd_size =construct_command (ace, &cmd, cmd_id);
There doesn't seem to be a need to take the lock during construct_command(). Maybe just during mei_ace_send()? You could also acquire and release the lock there I guess.
Quoted 96 lines
drivers/media/pci/intel/ivsc/mei_ace.c new file +96
-0+if (cmd_size >=0 ) + ret =mei_ace_send (ace, &cmd, cmd_size, false); +else + ret = cmd_size; +mutex_unlock (&ace->lock); + +return ret; +} + +/* the first command downloaded to firmware */ +static inline int ace_get_firmware_id (struct mei_ace *ace) +{ +struct ace_cmd cmd; +int cmd_size; +int ret; + + cmd_size =construct_command (ace, &cmd,ACE_GET_FW_ID ); +if (cmd_size >=0 ) + ret =mei_ace_send (ace, &cmd, cmd_size, true); +else + ret = cmd_size; + +return ret; +} + +static void handle_command_response (struct mei_ace *ace, +struct ace_notif *resp,int len) +{ +union ace_notif_hdr *hdr = &resp->hdr; + +switch (hdr->response.cmd_id) { +case ACE_SWITCH_CAMERA_TO_IVSC : +case ACE_SWITCH_CAMERA_TO_HOST : +memcpy (&ace->cmd_response, resp, len); +complete (&ace->cmd_completion); +break ; +case ACE_GET_FW_ID : +break ; +default : +break ; + } +} + +static void handle_command_ack (struct mei_ace *ace, +struct ace_notif *ack,int len) +{ +union ace_notif_hdr *hdr = &ack->hdr; + +switch (hdr->ack.cmd_id) { +case ACE_GET_FW_ID : + ace->firmware_id = ack->cont.firmware_id; + fallthrough; +case ACE_SWITCH_CAMERA_TO_IVSC : +case ACE_SWITCH_CAMERA_TO_HOST : +memcpy (&ace->cmd_ack, ack, len); +complete (&ace->cmd_completion); +break ; +default : +break ; + } +} + +/* callback for receive */ +static void mei_ace_rx (struct mei_cl_device *cldev) +{ +struct mei_ace *ace =mei_cldev_get_drvdata (cldev); +struct ace_notif event; +union ace_notif_hdr *hdr = &event.hdr; +int ret; + + ret =mei_cldev_recv (cldev, (u8 *)&event,sizeof (event)); +if (ret <0 ) { +dev_err (&cldev->dev,"recv error: %d\n" , ret); +return ; + } + +if (hdr->event.ack) { +handle_command_ack (ace, &event, ret); +return ; + } + +switch (hdr->event.event_type) { +case ACE_CMD_RESPONSE : +handle_command_response (ace, &event, ret); +break ; +case ACE_FW_READY : +/* +* firmware ready notification sent to driver +* after HECI client connected with firmware. +*/ +dev_dbg (&cldev->dev,"firmware ready\n" ); +break ; +default : +break ; + } +}
drivers/media/pci/intel/ivsc/mei_ace.c line 394
Quoted 10 lines
drivers/media/pci/intel/ivsc/mei_ace.c new file +10
-0+ +static int mei_ace_setup_dev_link (struct mei_ace *ace) +{ +struct device *dev = &ace->cldev->dev; +uuid_le uuid =MEI_CSI_UUID ; +struct acpi_device *adev; +struct device *csi_dev; +char name[64 ]; + +sprintf (name,"%s-%pUl" ,dev_name (dev->parent), &uuid);
snprintf(), please.
drivers/media/pci/intel/ivsc/mei_ace.c line 400
Quoted 6 lines
drivers/media/pci/intel/ivsc/mei_ace.c new file +6
-0+ csi_dev =device_find_child_by_name (dev->parent, name); +if (!csi_dev) +return -EPROBE_DEFER ; + +/* sensor's ACPI _DEP is mei bus device */ + adev =acpi_dev_get_next_consumer_dev (ACPI_COMPANION (dev->parent),NULL );
Nice.
Please also run this on the patchset:
./scripts/checkpatch.pl --strict --max-line-length=80
drivers/media/pci/intel/ivsc/mei_ace.c line 410
Quoted 10 lines
drivers/media/pci/intel/ivsc/mei_ace.c new file +10
-0+if (!adev) +return -EPROBE_DEFER ; + +/* setup link between mei_ace and mei_csi */ + ace->csi_link =device_link_add (csi_dev, dev, +DL_FLAG_PM_RUNTIME |DL_FLAG_RPM_ACTIVE ); + +/* setup link between mei_ace and sensor */ + ace->sensor_link =device_link_add (&adev->dev, dev, +DL_FLAG_PM_RUNTIME |DL_FLAG_RPM_ACTIVE );
device_link_add() may fail.
Quoted 124 lines
drivers/media/pci/intel/ivsc/mei_ace.c new file +124
-0+ +return 0 ; +} + +static int mei_ace_probe (struct mei_cl_device *cldev, +const struct mei_cl_device_id *id) +{ +struct device *dev = &cldev->dev; +struct mei_ace *ace; +int ret; + + ace =devm_kzalloc (dev,sizeof (struct mei_ace ),GFP_KERNEL ); +if (!ace) +return -ENOMEM ; + + ace->cldev = cldev; + + ret =mei_ace_setup_dev_link (ace); +if (ret) +return ret; + +mutex_init (&ace->lock); +init_completion (&ace->cmd_completion); + +mei_cldev_set_drvdata (cldev, ace); + + ret =mei_cldev_enable (cldev); +if (ret <0 ) { +dev_err (dev,"mei_cldev_enable failed: %d\n" , ret); + goto destroy_mutex; + } + + ret =mei_cldev_register_rx_cb (cldev, mei_ace_rx); +if (ret) { +dev_err (dev,"event cb registration failed: %d\n" , ret); + goto err_disable; + } + + ret =ace_get_firmware_id (ace); +if (ret) { +dev_err (dev,"get firmware id failed: %d\n" , ret); + goto err_disable; + } + +pm_runtime_set_active (dev); +pm_runtime_enable (dev); + +acpi_dev_clear_dependencies (ACPI_COMPANION (dev->parent)); + +return 0 ; + +err_disable: +mei_cldev_disable (cldev); +destroy_mutex: +mutex_destroy (&ace->lock); + +device_link_del (ace->csi_link); +device_link_del (ace->sensor_link); + +return ret; +} + +static void mei_ace_remove (struct mei_cl_device *cldev) +{ +struct mei_ace *ace =mei_cldev_get_drvdata (cldev); + +device_link_del (ace->csi_link); +device_link_del (ace->sensor_link); + +pm_runtime_disable (&cldev->dev); +pm_runtime_set_suspended (&cldev->dev); + +mutex_destroy (&ace->lock); +} + +static int __maybe_unused mei_ace_runtime_suspend (struct device *dev) +{ +struct mei_ace *ace =dev_get_drvdata (dev); + +return ace_set_camera_owner (ace,ACE_SWITCH_CAMERA_TO_IVSC ); +} + +static int __maybe_unused mei_ace_runtime_resume (struct device *dev) +{ +struct mei_ace *ace =dev_get_drvdata (dev); + +return ace_set_camera_owner (ace,ACE_SWITCH_CAMERA_TO_HOST ); +} + +static const struct dev_pm_ops mei_ace_pm_ops = { +SET_RUNTIME_PM_OPS (mei_ace_runtime_suspend, + mei_ace_runtime_resume, +NULL ) +}; + +#define MEI_ACE_UUID UUID_LE(0x5DB76CF6, 0x0A68, 0x4ED6, \ + 0x9B, 0x78, 0x03, 0x61, 0x63, 0x5E, 0x24, 0x47) + +static const struct mei_cl_device_id mei_ace_tbl[] = { + {MEI_ACE_DRIVER_NAME ,MEI_ACE_UUID ,MEI_CL_VERSION_ANY }, + +/* required last entry */ + { } +}; +MODULE_DEVICE_TABLE (mei, mei_ace_tbl); + +static struct mei_cl_driver mei_ace_driver = { + .id_table = mei_ace_tbl, + .name =MEI_ACE_DRIVER_NAME , + + .probe = mei_ace_probe, + .remove = mei_ace_remove, + + .driver = { + .pm = &mei_ace_pm_ops, + }, +}; + +module_mei_cl_driver (mei_ace_driver); + +MODULE_AUTHOR ("Wentong Wu <[email protected]>" ); +MODULE_AUTHOR ("Zhifeng Wang <[email protected]>" ); +MODULE_DESCRIPTION ("Device driver for IVSC ACE" ); +MODULE_LICENSE ("GPL" );
Signature · 3 lines
Kind regards, Sakari Ailus
RE: [PATCH v3 2/3] media: pci: intel: ivsc: Add ACE submodule
Hi Sakari,
Thanks
Quoted 64 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Wednesday, March 29, 2023 4:04 PMHi Wentong,
Quoted 57 lines · Wentong Wu
ACE is a submodule of IVSC which controls camera sensor's ownership, belonging to host or IVSC. When IVSC owns camera sensor, it is for algorithm computing. When host wants to control camera sensor, ACE module needs to be informed of ownership with defined interface.
The interface is via MEI. There is a separate MEI UUID, which this driver uses to enumerate.
To switch ownership of camera sensor between IVSC and host, the caller specifies the defined ownership information which will be sent to firmware by sending MEI command.
Device
link(device_link_add)is used to set the right camera sensor ownership before accessing the sensor via I²C. WithDL_FLAG_PM_RUNTIMEandDL_FLAG_RPM_ACTIVE, the supplier device will be PM runtime resumed before the consumer(camera sensor).
So use runtime PM callbacks to transfer the ownership between host and IVSC.Signed-off-by Wentong Wu <[email protected]>--- drivers/media/pci/intel/ivsc/Makefile | 3 + drivers/media/pci/intel/ivsc/mei_ace.c | 534 +++++++++++++++++++++++++++++++++ 2 files changed, 537 insertions(+) create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.cdrivers/media/pci/intel/ivsc/Makefile +0
-0@@ -5,3 +5,6 @@ obj-$(CONFIG_INTEL_VSC) += ivsc-csi.o ivsc-csi-y += mei_csi.oivsc-csi-y += csi_bridge.o
drivers/media/pci/intel/ivsc/Makefile +2
-0+ +obj-$(CONFIG_INTEL_VSC) += ivsc-ace.o ivsc-ace-y += mei_ace.odrivers/media/pci/intel/ivsc/mei_ace.c new file +13
-0@@ -0,0 +1,534 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright (C) 2023 Intel Corporation. All rights reserved. +* Intel Visual Sensing Controller ACE Linux driver */ + +/* +* To set ownership of camera sensor, there is specific command, +which +* is sent via MEI protocol. That's a two-step scheme where the +firmware +* first acks receipt of the command and later responses the command +was +* executed. The command sending function uses "completion" as the
Quoted 8 lines
drivers/media/pci/intel/ivsc/mei_ace.c line 18
Quoted 5 lines
drivers/media/pci/intel/ivsc/mei_ace.c new file +5
-0+ * synchronization mechanism. The notification for command is +received + * via a mei callback which wakes up the caller. There can be only +one + * outstanding command at a time.Could you document the dependencies in the sensor initialisation (ace + csi)
This is about ownership instead of dependency, if host sensor driver configure sensor with ownership on IVSC, probably it will be changed by firmware somehow.
Quoted 2 lines
and how the sensor is powered (i.e. no direct control of the PMIC from the host?)?
The power line is directly connected to IVSC instead of host, when ownership switched to host, sensor is already powered up by firmware.
Quoted 268 lines
This doesn't seem like a bad place to do it.
Quoted 20 lines
drivers/media/pci/intel/ivsc/mei_ace.c new file +20
-0+*/ + +#include <linux/acpi.h> +#include <linux/completion.h> +#include <linux/delay.h> +#include <linux/kernel.h> +#include <linux/mei_cl_bus.h> +#include <linux/module.h> +#include <linux/mutex.h> +#include <linux/pm_runtime.h> +#include <linux/slab.h> +#include <linux/uuid.h> + +#define MEI_ACE_DRIVER_NAME "ivsc_ace" + +/* indicating driver message */ +#define ACE_DRV_MSG 1 +/* indicating set command */ +#define ACE_CMD_SET 4 +/* command timeout determined experimentally */ drivers/media/pci/intel/ivsc/mei_ace.c
Quoted 10 lines
drivers/media/pci/intel/ivsc/mei_ace.c new file +10
-0+#define ACE_CMD_TIMEOUT (5 * HZ) +/* indicating the first command block */ +#define ACE_CMD_INIT_BLOCK 1 +/* indicating the last command block */ +#define ACE_CMD_FINAL_BLOCK 1 +/* size of camera status notification content */ +#define ACE_CAMERA_STATUS_SIZE 5 + +/* UUID used to get firmware id */ +#define ACE_GET_FW_ID_UUID UUID_LE(0x6167DCFB, 0x72F1, 0x4584,
0xBF, \Quoted 235 lines
drivers/media/pci/intel/ivsc/mei_ace.c new file +235
-0+ 0xE3, 0x84, 0x17, 0x71, 0xAA, 0x79, 0x0B) + +/* UUID used to get csi device */ +#define MEI_CSI_UUID UUID_LE(0x92335FCF, 0x3203, 0x4472, \ + 0xAF, 0x93, 0x7b, 0x44, 0x53, 0xAC, 0x29, 0xDA) + +/* identify firmware event type */ +enum ace_event_type { +/* firmware ready */ +ACE_FW_READY =0x8 , + +/* command response */ +ACE_CMD_RESPONSE =0x10 , +}; + +/* identify camera sensor ownership */ enum ace_camera_owner { +ACE_CAMERA_IVSC , +ACE_CAMERA_HOST , +}; + +/* identify the command id supported by firmware IPC */ enum +ace_cmd_id { +/* used to switch camera sensor to host */ +ACE_SWITCH_CAMERA_TO_HOST =0x13 , + +/* used to switch camera sensor to IVSC */ +ACE_SWITCH_CAMERA_TO_IVSC =0x14 , + +/* used to get firmware id */ +ACE_GET_FW_ID =0x1A , +}; + +/* ACE command header structure */ +struct ace_cmd_hdr { +u32 firmware_id :16 ; +u32 instance_id :8 ; +u32 type :5 ; +u32 rsp :1 ; +u32 msg_tgt :1 ; +u32 _hw_rsvd_1 :1 ; +u32 param_size :20 ; +u32 cmd_id :8 ; +u32 final_block :1 ; +u32 init_block :1 ; +u32 _hw_rsvd_2 :2 ; +} __packed; + +/* ACE command parameter structure */ union ace_cmd_param { +uuid_le uuid; +u32 param; +}; + +/* ACE command structure */ +struct ace_cmd { +struct ace_cmd_hdr hdr; +union ace_cmd_param param; +} __packed; + +/* ACE notification header */ +union ace_notif_hdr { +struct _confirm { +u32 status :24 ; +u32 type :5 ; +u32 rsp :1 ; +u32 msg_tgt :1 ; +u32 _hw_rsvd_1 :1 ; +u32 param_size :20 ; +u32 cmd_id :8 ; +u32 final_block :1 ; +u32 init_block :1 ; +u32 _hw_rsvd_2 :2 ; + } __packed ack; + +struct _event { +u32 rsvd1 :16 ; +u32 event_type :8 ; +u32 type :5 ; +u32 ack :1 ; +u32 msg_tgt :1 ; +u32 _hw_rsvd_1 :1 ; +u32 rsvd2 :30 ; +u32 _hw_rsvd_2 :2 ; + } __packed event; + +struct _response { +u32 event_id :16 ; +u32 notif_type :8 ; +u32 type :5 ; +u32 rsp :1 ; +u32 msg_tgt :1 ; +u32 _hw_rsvd_1 :1 ; +u32 event_data_size :16 ; +u32 request_target :1 ; +u32 request_type :5 ; +u32 cmd_id :8 ; +u32 _hw_rsvd_2 :2 ; + } __packed response; +}; + +/* ACE notification content */ +union ace_notif_cont { +u16 firmware_id; +u8 state_notif; +u8 camera_status[ACE_CAMERA_STATUS_SIZE ]; +}; + +/* ACE notification structure */ +struct ace_notif { +union ace_notif_hdr hdr; +union ace_notif_cont cont; +} __packed; + +struct mei_ace { +struct mei_cl_device *cldev; + +/* command ack */ +struct ace_notif cmd_ack; +/* command response */ +struct ace_notif cmd_response; +/* used to wait for command ack and response */ +struct completion cmd_completion; +/* lock used to prevent multiple call to ace */ +struct mutex lock; + +/* used to construct command */ +u16 firmware_id; + +/* runtime PM link from ace to csi */ +struct device_link *csi_link; +/* runtime PM link from ace to sensor */ +struct device_link *sensor_link; +}; + +static inline void init_cmd_hdr (struct ace_cmd_hdr *hdr) { +memset (hdr,0 ,sizeof (struct ace_cmd_hdr )); + + hdr->type =ACE_CMD_SET ; + hdr->msg_tgt =ACE_DRV_MSG ; + hdr->init_block =ACE_CMD_INIT_BLOCK ; + hdr->final_block =ACE_CMD_FINAL_BLOCK ; } + +static int construct_command (struct mei_ace *ace,struct ace_cmd *cmd, +enum ace_cmd_id cmd_id) +{ +union ace_cmd_param *param = &cmd->param; +struct ace_cmd_hdr *hdr = &cmd->hdr; + +init_cmd_hdr (hdr); + + hdr->cmd_id = cmd_id; +switch (cmd_id) { +case ACE_GET_FW_ID : + param->uuid =ACE_GET_FW_ID_UUID ; + hdr->param_size =sizeof (param->uuid); +break ; +case ACE_SWITCH_CAMERA_TO_IVSC : + param->param =0 ; + hdr->firmware_id = ace->firmware_id; + hdr->param_size =sizeof (param->param); +break ; +case ACE_SWITCH_CAMERA_TO_HOST : + hdr->firmware_id = ace->firmware_id; +break ; +default : +return -EINVAL ; + } + +return hdr->param_size +sizeof (cmd->hdr); } + +/* send a command to firmware and mutex must be held by caller */ +static int mei_ace_send (struct mei_ace *ace,struct ace_cmd *cmd, +size_t len,bool only_ack) +{ +union ace_notif_hdr *resp_hdr = &ace->cmd_response.hdr; +union ace_notif_hdr *ack_hdr = &ace->cmd_ack.hdr; +struct ace_cmd_hdr *cmd_hdr = &cmd->hdr; +int ret; + +reinit_completion (&ace->cmd_completion); + + ret =mei_cldev_send (ace->cldev, (u8 *)cmd, len); +if (ret <0 ) + goto out; + + ret =wait_for_completion_killable_timeout (&ace->cmd_completion, +ACE_CMD_TIMEOUT ); +if (ret <0 ) { + goto out; + }else if (!ret) { + ret = -ETIMEDOUT ; + goto out; + } + +if (ack_hdr->ack.cmd_id != cmd_hdr->cmd_id) { + ret = -EINVAL ; + goto out; + } + +/* command ack status */ + ret = ack_hdr->ack.status; +if (ret) { + ret = -EIO ; + goto out; + } + +if (only_ack) + goto out; + + ret =wait_for_completion_killable_timeout (&ace->cmd_completion, +ACE_CMD_TIMEOUT ); +if (ret <0 ) { + goto out; + }else if (!ret) { + ret = -ETIMEDOUT ; + goto out; + } + +if (resp_hdr->response.cmd_id != cmd_hdr->cmd_id) + ret = -EINVAL ; + +out: +return ret; +} + +static int ace_set_camera_owner (struct mei_ace *ace,enum +ace_camera_owner owner) { +enum ace_cmd_id cmd_id; +struct ace_cmd cmd; +int cmd_size; +int ret; + +if (owner ==ACE_CAMERA_IVSC ) + cmd_id =ACE_SWITCH_CAMERA_TO_IVSC ; +else + cmd_id =ACE_SWITCH_CAMERA_TO_HOST ;
Quoted 8 lines
drivers/media/pci/intel/ivsc/mei_ace.c line 288
Quoted 4 lines
drivers/media/pci/intel/ivsc/mei_ace.c new file +4
-0+ +mutex_lock (&ace->lock); + + cmd_size =construct_command (ace, &cmd, cmd_id);There doesn't seem to be a need to take the lock during
construct_command(). Maybe just duringmei_ace_send()? You could also acquire and release the lock there I guess.
Thanks
Quoted 96 lines
Quoted 96 lines
drivers/media/pci/intel/ivsc/mei_ace.c new file +95
-0+if (cmd_size >=0 ) + ret =mei_ace_send (ace, &cmd, cmd_size, false); +else + ret = cmd_size; +mutex_unlock (&ace->lock); + +return ret; +} + +/* the first command downloaded to firmware */ static inline int +ace_get_firmware_id (struct mei_ace *ace) { +struct ace_cmd cmd; +int cmd_size; +int ret; + + cmd_size =construct_command (ace, &cmd,ACE_GET_FW_ID ); +if (cmd_size >=0 ) + ret =mei_ace_send (ace, &cmd, cmd_size, true); +else + ret = cmd_size; + +return ret; +} + +static void handle_command_response (struct mei_ace *ace, +struct ace_notif *resp,int len) { +union ace_notif_hdr *hdr = &resp->hdr; + +switch (hdr->response.cmd_id) { +case ACE_SWITCH_CAMERA_TO_IVSC : +case ACE_SWITCH_CAMERA_TO_HOST : +memcpy (&ace->cmd_response, resp, len); +complete (&ace->cmd_completion); +break ; +case ACE_GET_FW_ID : +break ; +default : +break ; + } +} + +static void handle_command_ack (struct mei_ace *ace, +struct ace_notif *ack,int len) { +union ace_notif_hdr *hdr = &ack->hdr; + +switch (hdr->ack.cmd_id) { +case ACE_GET_FW_ID : + ace->firmware_id = ack->cont.firmware_id; + fallthrough; +case ACE_SWITCH_CAMERA_TO_IVSC : +case ACE_SWITCH_CAMERA_TO_HOST : +memcpy (&ace->cmd_ack, ack, len); +complete (&ace->cmd_completion); +break ; +default : +break ; + } +} + +/* callback for receive */ +static void mei_ace_rx (struct mei_cl_device *cldev) { +struct mei_ace *ace =mei_cldev_get_drvdata (cldev); +struct ace_notif event; +union ace_notif_hdr *hdr = &event.hdr; +int ret; + + ret =mei_cldev_recv (cldev, (u8 *)&event,sizeof (event)); +if (ret <0 ) { +dev_err (&cldev->dev,"recv error: %d\n" , ret); +return ; + } + +if (hdr->event.ack) { +handle_command_ack (ace, &event, ret); +return ; + } + +switch (hdr->event.event_type) { +case ACE_CMD_RESPONSE : +handle_command_response (ace, &event, ret); +break ; +case ACE_FW_READY : +/* +* firmware ready notification sent to driver +* after HECI client connected with firmware. +*/ +dev_dbg (&cldev->dev,"firmware ready\n" ); +break ; +default : +break ; + } +} + +static int mei_ace_setup_dev_link (struct mei_ace *ace) { +struct device *dev = &ace->cldev->dev;
Quoted 8 lines
drivers/media/pci/intel/ivsc/mei_ace.c line 394
Quoted 6 lines
drivers/media/pci/intel/ivsc/mei_ace.c new file +6
-0+uuid_le uuid =MEI_CSI_UUID ; +struct acpi_device *adev; +struct device *csi_dev; +char name[64 ]; + +sprintf (name,"%s-%pUl" ,dev_name (dev->parent), &uuid);
snprintf(), please.
Thanks
Quoted 12 lines
Quoted 9 lines
drivers/media/pci/intel/ivsc/mei_ace.c new file +6
-0+ csi_dev =device_find_child_by_name (dev->parent, name); +if (!csi_dev) +return -EPROBE_DEFER ; + +/* sensor's ACPI _DEP is mei bus device */ + adev = acpi_dev_get_next_consumer_dev(ACPI_COMPANION(dev-parent),
+NULL);Nice.
Quoted 3 lines
Please also run this on the patchset:
./scripts/checkpatch.pl --strict --max-line-length=80
ack
Quoted 8 lines
drivers/media/pci/intel/ivsc/mei_ace.c
Quoted 7 lines
drivers/media/pci/intel/ivsc/mei_ace.c new file +6
-0+if (!adev) +return -EPROBE_DEFER ; + +/* setup link between mei_ace and mei_csi */ + ace->csi_link = device_link_add(csi_dev, dev, + DL_FLAG_PM_RUNTIME |
DL_FLAG_RPM_ACTIVE);
Quoted 7 lines
drivers/media/pci/intel/ivsc/mei_ace.c
Quoted 4 lines
drivers/media/pci/intel/ivsc/mei_ace.c new file +4
-0+ +/* setup link between mei_ace and sensor */ + ace->sensor_link = device_link_add(&adev->dev, dev, + DL_FLAG_PM_RUNTIME |
DL_FLAG_RPM_ACTIVE);
device_link_add()may fail.
Thanks
Quoted 127 lines
Quoted 75 lines
drivers/media/pci/intel/ivsc/mei_ace.c new file +74
-0+ +return 0 ; +} + +static int mei_ace_probe (struct mei_cl_device *cldev, +const struct mei_cl_device_id *id) { +struct device *dev = &cldev->dev; +struct mei_ace *ace; +int ret; + + ace =devm_kzalloc (dev,sizeof (struct mei_ace ),GFP_KERNEL ); +if (!ace) +return -ENOMEM ; + + ace->cldev = cldev; + + ret =mei_ace_setup_dev_link (ace); +if (ret) +return ret; + +mutex_init (&ace->lock); +init_completion (&ace->cmd_completion); + +mei_cldev_set_drvdata (cldev, ace); + + ret =mei_cldev_enable (cldev); +if (ret <0 ) { +dev_err (dev,"mei_cldev_enable failed: %d\n" , ret); + goto destroy_mutex; + } + + ret =mei_cldev_register_rx_cb (cldev, mei_ace_rx); +if (ret) { +dev_err (dev,"event cb registration failed: %d\n" , ret); + goto err_disable; + } + + ret =ace_get_firmware_id (ace); +if (ret) { +dev_err (dev,"get firmware id failed: %d\n" , ret); + goto err_disable; + } + +pm_runtime_set_active (dev); +pm_runtime_enable (dev); + +acpi_dev_clear_dependencies (ACPI_COMPANION (dev->parent)); + +return 0 ; + +err_disable: +mei_cldev_disable (cldev); +destroy_mutex: +mutex_destroy (&ace->lock); + +device_link_del (ace->csi_link); +device_link_del (ace->sensor_link); + +return ret; +} + +static void mei_ace_remove (struct mei_cl_device *cldev) { +struct mei_ace *ace =mei_cldev_get_drvdata (cldev); + +device_link_del (ace->csi_link); +device_link_del (ace->sensor_link); + +pm_runtime_disable (&cldev->dev); +pm_runtime_set_suspended (&cldev->dev); + +mutex_destroy (&ace->lock); +} + +static int __maybe_unused mei_ace_runtime_suspend (struct device *dev)drivers/media/pci/intel/ivsc/mei_ace.c
Quoted 10 lines
drivers/media/pci/intel/ivsc/mei_ace.c new file +10
-0+{ +struct mei_ace *ace =dev_get_drvdata (dev); + +return ace_set_camera_owner (ace,ACE_SWITCH_CAMERA_TO_IVSC ); } + +static int __maybe_unused mei_ace_runtime_resume (struct device *dev) +{ +struct mei_ace *ace =dev_get_drvdata (dev); + + return ace_set_camera_owner(ace,
ACE_SWITCH_CAMERA_TO_HOST); }Quoted 26 lines
drivers/media/pci/intel/ivsc/mei_ace.c new file +26
-0+ +static const struct dev_pm_ops mei_ace_pm_ops = { +SET_RUNTIME_PM_OPS (mei_ace_runtime_suspend, + mei_ace_runtime_resume, +NULL ) +}; + +#define MEI_ACE_UUID UUID_LE(0x5DB76CF6, 0x0A68, 0x4ED6, \ + 0x9B, 0x78, 0x03, 0x61, 0x63, 0x5E, 0x24, 0x47) + +static const struct mei_cl_device_id mei_ace_tbl[] = { + {MEI_ACE_DRIVER_NAME ,MEI_ACE_UUID ,MEI_CL_VERSION_ANY }, + +/* required last entry */ + { } +}; +MODULE_DEVICE_TABLE (mei, mei_ace_tbl); + +static struct mei_cl_driver mei_ace_driver = { + .id_table = mei_ace_tbl, + .name =MEI_ACE_DRIVER_NAME , + + .probe = mei_ace_probe, + .remove = mei_ace_remove, + + .driver = {drivers/media/pci/intel/ivsc/mei_ace.c line 534
Quoted 10 lines
drivers/media/pci/intel/ivsc/mei_ace.c new file +10
-0+ .pm = &mei_ace_pm_ops, + }, +}; + +module_mei_cl_driver (mei_ace_driver); + +MODULE_AUTHOR ("Wentong Wu <[email protected]>" ); +MODULE_AUTHOR ("Zhifeng Wang <[email protected]>" ); +MODULE_DESCRIPTION ("Device driver for IVSC ACE" ); +MODULE_LICENSE ("GPL" );--
Kind regards,Sakari Ailus
Sakari Ailus
maintainerRe: [PATCH v3 2/3] media: pci: intel: ivsc: Add ACE submodule
Hi Wentong,
Quoted 77 lines · Wentong
Hi Sakari,
Thanks
Quoted 64 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Wednesday, March 29, 2023 4:04 PMHi Wentong,
Quoted 57 lines · Wentong Wu
ACE is a submodule of IVSC which controls camera sensor's ownership, belonging to host or IVSC. When IVSC owns camera sensor, it is for algorithm computing. When host wants to control camera sensor, ACE module needs to be informed of ownership with defined interface.
The interface is via MEI. There is a separate MEI UUID, which this driver uses to enumerate.
To switch ownership of camera sensor between IVSC and host, the caller specifies the defined ownership information which will be sent to firmware by sending MEI command.
Device
link(device_link_add)is used to set the right camera sensor ownership before accessing the sensor via I²C. WithDL_FLAG_PM_RUNTIMEandDL_FLAG_RPM_ACTIVE, the supplier device will be PM runtime resumed before the consumer(camera sensor).
So use runtime PM callbacks to transfer the ownership between host and IVSC.Signed-off-by Wentong Wu <[email protected]>--- drivers/media/pci/intel/ivsc/Makefile | 3 + drivers/media/pci/intel/ivsc/mei_ace.c | 534 +++++++++++++++++++++++++++++++++ 2 files changed, 537 insertions(+) create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.cdrivers/media/pci/intel/ivsc/Makefile +0
-0@@ -5,3 +5,6 @@ obj-$(CONFIG_INTEL_VSC) += ivsc-csi.o ivsc-csi-y += mei_csi.oivsc-csi-y += csi_bridge.o
drivers/media/pci/intel/ivsc/Makefile +2
-0+ +obj-$(CONFIG_INTEL_VSC) += ivsc-ace.o ivsc-ace-y += mei_ace.odrivers/media/pci/intel/ivsc/mei_ace.c new file +13
-0@@ -0,0 +1,534 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright (C) 2023 Intel Corporation. All rights reserved. +* Intel Visual Sensing Controller ACE Linux driver */ + +/* +* To set ownership of camera sensor, there is specific command, +which +* is sent via MEI protocol. That's a two-step scheme where the +firmware +* first acks receipt of the command and later responses the command +was +* executed. The command sending function uses "completion" as the Quoted 8 lines
drivers/media/pci/intel/ivsc/mei_ace.c line 18
Quoted 5 lines
drivers/media/pci/intel/ivsc/mei_ace.c new file +5
-0+ * synchronization mechanism. The notification for command is +received + * via a mei callback which wakes up the caller. There can be only +one + * outstanding command at a time.Could you document the dependencies in the sensor initialisation (ace + csi)
Quoted 8 lines
This is about ownership instead of dependency, if host sensor driver configure sensor with ownership on IVSC, probably it will be changed by firmware somehow.
Quoted 2 lines
and how the sensor is powered (i.e. no direct control of the PMIC from the host?)?
The power line is directly connected to IVSC instead of host, when ownership switched to host, sensor is already powered up by firmware.
Good. Please also document this in the comment.
Signature · 3 lines
Kind regards, Sakari Ailus
PATCH v3 3/3 ACPI: delay enumeration of devices with a _DEP pointing to INTC1059 device
In reply to Wentong WuInside IVSC, switching ownership requires an interface with two different hardware modules, ACE and CSI. The software interface to these modules is based on Intel MEI framework. Usually mei client devices are dynamically created, so the info of consumers depending on mei client devices is not present in the firmware tables.
This causes problems with the probe ordering with respect to drivers for consumers of these mei client devices. But on these camera sensor devices, the ACPI nodes describing the sensors all have a _DEP dependency on the matching INTC1059 ACPI device, so adding INTC1059 to acpi_honor_dep_ids allows solving the probe-ordering problem by delaying the enumeration of ACPI-devices which have a _DEP dependency on an INTC1059 device.
--- drivers/acpi/scan.c | 1 + 1 file changed, 1 insertion(+)
drivers/acpi/scan.c +1 -0
@@ -796,6 +796,7 @@ static const char * const acpi_ignore_dep_ids[] = {/* List of HIDs for which we honor deps of matching ACPI devs, when checking _DEP lists. */ static const char *const acpi_honor_dep_ids[] = {"INT3472" ,/* Camera sensor PMIC / clk and regulator info */ +"INTC1059" ,NULL };
Signature · 1 line
2.7.4
Sakari Ailus
maintainerRe: [PATCH v3 3/3] ACPI: delay enumeration of devices with a _DEP pointing to INTC1059 device
Hi Wentong,
Quoted 5 lines · Wentong Wu
Inside IVSC, switching ownership requires an interface with two different hardware modules, ACE and CSI. The software interface to these modules is based on Intel MEI framework. Usually mei client devices are dynamically created, so the info of consumers depending on mei client devices is not present in the firmware tables.
Ouch.
Quoted 7 lines
This causes problems with the probe ordering with respect to drivers for consumers of these mei client devices. But on these camera sensor devices, the ACPI nodes describing the sensors all have a
_DEPdependency on the matching INTC1059 ACPI device, so adding INTC1059 toacpi_honor_dep_idsallows solving the probe-ordering problem by delaying the enumeration of ACPI-devices which have a_DEPdependency on an INTC1059 device.
What does the INTC1059 device represent?
Signature · 1 line
Sakari Ailus
RE: [PATCH v3 3/3] ACPI: delay enumeration of devices with a _DEP pointing to INTC1059 device
Quoted 7 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>Hi Wentong,
Quoted 8 lines
Quoted 5 lines · Wentong Wu
Inside IVSC, switching ownership requires an interface with two different hardware modules, ACE and CSI. The software interface to these modules is based on Intel MEI framework. Usually mei client devices are dynamically created, so the info of consumers depending on mei client devices is not present in the firmware tables.
Ouch.
MEI client devices are not ACPI devices in our case.
Quoted 11 lines
Quoted 8 lines
This causes problems with the probe ordering with respect to drivers for consumers of these mei client devices. But on these camera sensor devices, the ACPI nodes describing the sensors all have a
_DEPdependency on the matching INTC1059 ACPI device, so adding INTC1059 toacpi_honor_dep_idsallows solving the probe-ordering problem by delaying the enumeration of ACPI-devices which have a_DEPdependency onan INTC1059 device.
What does the INTC1059 device represent?
This is the IVSC device on which camera sensor has _DEP
Device (CVFD)
{
Name (_HID, "INTC1059") // _HID: Hardware ID
Name (_DDN, "Intel CVF Device") // _DDN: DOS Device Name
Method (SID, 1, Serialized)
{
Return (HCID (One))
}
}
Device (LNK1)
{
Name (PNIO, Package (0x01)
{
"\\_SB.PC00.SPI1.SPFD.CVFD"
})
Name (PUSB, Package (0x02)
{
"\\_SB.PC00.SPI1.SPFD.CVFD",
"\\_SB.PC00.XHCI.RHUB.HS08.VIC0"
})
Method (_STA, 0, NotSerialized) // _STA: Status
{
If (L1EN)
{
Return (0x0F)
}
Else
{
Return (Zero)
}
}
Method (_DEP, 0, NotSerialized) // _DEP: Dependencies
{
If (L1EN)
{
If ((CVFS == 0x02))
{
Return (PUSB) /* \_SB_.PC00.LNK1.PUSB */
}
If ((CVFS == One))
{
Return (PNIO) /* \_SB_.PC00.LNK1.PNIO */
}
Else
{
Return (CDEP (L1CL, L1BS))
}
}
Else
{
Return (Package (0x01)
{
PC00
})
}
}Quoted 3 lines
--
Sakari Ailus
Sakari Ailus
maintainerRe: [PATCH v3 3/3] ACPI: delay enumeration of devices with a _DEP pointing to INTC1059 device
Hi Wentong,
Quoted 32 lines · Wentong
Quoted 7 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>Hi Wentong,
Quoted 8 lines
Quoted 5 lines · Wentong Wu
Inside IVSC, switching ownership requires an interface with two different hardware modules, ACE and CSI. The software interface to these modules is based on Intel MEI framework. Usually mei client devices are dynamically created, so the info of consumers depending on mei client devices is not present in the firmware tables.
Ouch.
MEI client devices are not ACPI devices in our case.
Quoted 11 lines
Quoted 8 lines
This causes problems with the probe ordering with respect to drivers for consumers of these mei client devices. But on these camera sensor devices, the ACPI nodes describing the sensors all have a
_DEPdependency on the matching INTC1059 ACPI device, so adding INTC1059 toacpi_honor_dep_idsallows solving the probe-ordering problem by delaying the enumeration of ACPI-devices which have a_DEPdependency onan INTC1059 device.
What does the INTC1059 device represent?
This is the IVSC device on which camera sensor has
_DEP
Quoted 9 lines
Device (CVFD)
{
Name (_HID, "INTC1059") // _HID: Hardware ID
Name (_DDN, "Intel CVF Device") // _DDN: DOS Device Name
Method (SID, 1, Serialized)
{
Return (HCID (One))
}
}Should this be interpreted that if you have a _DEP to this device, then the IVSC is between the sensor and the IPU?
Signature · 3 lines
Kind regards, Sakari Ailus
RE: [PATCH v3 3/3] ACPI: delay enumeration of devices with a _DEP pointing to INTC1059 device
Quoted 52 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Monday, March 27, 2023 3:33 PMHi Wentong,
Quoted 33 lines · Wentong
Quoted 7 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>Hi Wentong,
Quoted 8 lines
Quoted 5 lines · Wentong Wu
Inside IVSC, switching ownership requires an interface with two different hardware modules, ACE and CSI. The software interface to these modules is based on Intel MEI framework. Usually mei client devices are dynamically created, so the info of consumers depending on mei client devices is not present in the firmware tables.
Ouch.
MEI client devices are not ACPI devices in our case.
Quoted 9 lines
Quoted 9 lines
This causes problems with the probe ordering with respect to drivers for consumers of these mei client devices. But on these camera sensor devices, the ACPI nodes describing the sensors all have a
_DEPdependency on the matching INTC1059 ACPI device, so adding INTC1059 toacpi_honor_dep_idsallows solving the probe-ordering problem by delaying the enumeration of ACPI-devices which have a_DEPdependency onQuoted 3 lines
an INTC1059 device.
What does the INTC1059 device represent?
This is the IVSC device on which camera sensor has
_DEPQuoted 9 lines
Device (CVFD) { Name (_HID, "INTC1059") // _HID: Hardware ID Name (_DDN, "Intel CVF Device") // _DDN: DOS Device Name Method (SID, 1, Serialized) { Return (HCID (One)) } }
Quoted 2 lines
Should this be interpreted that if you have a
_DEPto this device, then the IVSC is between the sensor and the IPU?
Ok, I can explain more in next version patch set. Thanks
BR,
Wentong
Quoted 5 lines
--
Kind regards,Sakari Ailus
Sakari Ailus
maintainerHi Wentong,
Quoted 40 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up systemSince the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces exported via v4l2 sub-device.
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware modules are:
- ACE (Algorithm Context Engine): This module is for algorithm computing
when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, users are informed via v4l2 control API.
In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
Quoted 9 lines
Implementation:
There are two different drivers to handle ACE and CSI hardware modules inside IVSC.
-ivsc_csi: MEI client driver to send commands and receive notifications
from CSI module.
-ivsc_ace: MEI client driver to send commands and get status from ACE
module.
Interface is exposed via v4l2 sub-devcie APIs to acquire and release camera sensor and CSI-2 link.
Thanks for the update.
Could you elaborate the decision of keeping the csi_bridge entirely separate from the cio2_bridge (to be turned to ipu_bridge first)? Both are doing essentially the same and using the same data structures, aren't they?
Signature · 3 lines
Kind regards, Sakari Ailus
Quoted 63 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Monday, March 27, 2023 3:21 PMHi Wentong,
Quoted 22 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
systemSince the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces exported via v4l2 sub-device.
Quoted 4 lines
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware
modules are:
Quoted 3 lines
- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership ofcamera sensor.
Quoted 13 lines
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, users are informed via
v4l2 control API.In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
Quoted 9 lines
Implementation:
There are two different drivers to handle ACE and CSI hardware modules inside IVSC.
-ivsc_csi: MEI client driver to send commands and receive
notifications from CSI module.
-ivsc_ace: MEI client driver to send commands and get status from
ACE module.
Interface is exposed via v4l2 sub-devcie APIs to acquire and release camera sensor and CSI-2 link.
Quoted 5 lines
Thanks for the update.
Could you elaborate the decision of keeping the
csi_bridgeentirely separate from thecio2_bridge(to be turned toipu_bridgefirst)? Both are doing essentially the same and using the same data structures, aren't they?
yes, they're doing same thing to bridge the software nodes needed by v4l2, but they have different type devices(pci and mei_client device) and dependency. And they have same SSDB definition in DSDT, so the structures are almost same.
I have no idea what the ipu bridge would be like, but IVSC csi bridge can be configured via kconfig to enable/disable.
Quoted 5 lines
--
Kind regards,Sakari Ailus
Sakari Ailus
maintainerHi Wentong,
Quoted 65 lines · Wentong
Quoted 65 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Monday, March 27, 2023 3:21 PMHi Wentong,
Quoted 22 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
systemSince the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces exported via v4l2 sub-device.
Quoted 4 lines
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware
modules are:
Quoted 3 lines
- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership ofcamera sensor.
Quoted 13 lines
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, users are informed via
v4l2 control API.In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
Quoted 9 lines
Implementation:
There are two different drivers to handle ACE and CSI hardware modules inside IVSC.
-ivsc_csi: MEI client driver to send commands and receive
notifications from CSI module.
-ivsc_ace: MEI client driver to send commands and get status from
ACE module.
Interface is exposed via v4l2 sub-devcie APIs to acquire and release camera sensor and CSI-2 link.Thanks for the update.
Quoted 8 lines
Quoted 3 lines
Could you elaborate the decision of keeping the
csi_bridgeentirely separate from thecio2_bridge(to be turned toipu_bridgefirst)? Both are doing essentially the same and using the same data structures, aren't they?yes, they're doing same thing to bridge the software nodes needed by v4l2, but they have different type
devices(pci and mei_client device)and dependency. And they have same SSDB definition in DSDT, so the structures are almost same.
If there are differences, what are they?
What comes to cio2_bridge, the fact that it's related to a PCI device doesn't seem to matter after initialisation so it could as well work with struct device.
Quoted 3 lines
I have no idea what the ipu bridge would be like, but IVSC csi bridge can be configured via kconfig to enable/disable.
Please work out the details with Bingbu.
And please do wrap your lines at 74 or so.
Signature · 3 lines
Kind regards, Sakari Ailus
Quoted 80 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Monday, March 27, 2023 3:50 PMHi Wentong,
Quoted 24 lines · Wentong
Quoted 24 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Monday, March 27, 2023 3:21 PMHi Wentong,
Quoted 15 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
systemQuoted 6 lines
Quoted 6 lines
Quoted 6 lines
Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces exported
via v4l2 sub-device.
Quoted 35 lines
Quoted 35 lines
Quoted 6 lines
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware
modules are:
Quoted 4 lines
- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership ofcamera sensor.
Quoted 22 lines
- CSI (Camera Serial Interface): This module is used to route
camera sensor data either to IVSC or to host for IPU driver and application.IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, users are informed via
v4l2 control API.In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
Implementation:
There are two different drivers to handle ACE and CSI hardware modules inside IVSC.
-ivsc_csi: MEI client driver to send commands and receive
notifications from CSI module.
-ivsc_ace: MEI client driver to send commands and get status
from ACE module.
Interface is exposed via v4l2 sub-devcie APIs to acquire and release camera sensor and CSI-2 link.Quoted 5 lines
Quoted 5 lines
Thanks for the update.
Could you elaborate the decision of keeping the
csi_bridgeentirely separate from thecio2_bridge(to be turned toipu_bridgefirst)? Both are doing essentially the same and using the same data structures,
Quoted 9 lines
aren't they?
Quoted 5 lines
yes, they're doing same thing to bridge the software nodes needed by v4l2, but they have different type
devices(pci and mei_client device)and dependency. And they have same SSDB definition in DSDT, so the structures are almost same.If there are differences, what are they?
The properties of swnode are same, because we need same info to configure CSI2. But csi bridge can get sensor connected to IVSC by dependency info instead of just polling the only supported sensors for ipu. Also IVSC doesn't need vcm. And the topology of dependency is different.
Quoted 8 lines
What comes to
cio2_bridge, the fact that it's related to a PCI device doesn't seem to matter after initialisation so it could as well work with struct device.Quoted 3 lines
I have no idea what the ipu bridge would be like, but IVSC csi bridge can be configured via kconfig to enable/disable.
Please work out the details with Bingbu.
@[email protected] @Sakari Ailus Please share your design idea here. What the ipu bridge would be like? What's the responsibility of ipu bridge?
Quoted 7 lines
And please do wrap your lines at 74 or so.
--
Kind regards,Sakari Ailus
Sakari Ailus
maintainerHi Wentong,
Quoted 87 lines · Wentong
Quoted 87 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Monday, March 27, 2023 3:50 PMHi Wentong,
Quoted 24 lines · Wentong
Quoted 24 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Monday, March 27, 2023 3:21 PMHi Wentong,
Quoted 15 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up
systemQuoted 6 lines
Quoted 6 lines
Quoted 6 lines
Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces exported
via v4l2 sub-device.
Quoted 35 lines
Quoted 35 lines
Quoted 6 lines
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware
modules are:
Quoted 4 lines
- ACE (Algorithm Context Engine): This module is for algorithm
computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership ofcamera sensor.
Quoted 22 lines
- CSI (Camera Serial Interface): This module is used to route
camera sensor data either to IVSC or to host for IPU driver and application.IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, users are informed via
v4l2 control API.In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
Implementation:
There are two different drivers to handle ACE and CSI hardware modules inside IVSC.
-ivsc_csi: MEI client driver to send commands and receive
notifications from CSI module.
-ivsc_ace: MEI client driver to send commands and get status
from ACE module.
Interface is exposed via v4l2 sub-devcie APIs to acquire and release camera sensor and CSI-2 link.Quoted 5 lines
Quoted 5 lines
Thanks for the update.
Could you elaborate the decision of keeping the
csi_bridgeentirely separate from thecio2_bridge(to be turned toipu_bridgefirst)? Both are doing essentially the same and using the same data structures,aren't they?
Quoted 5 lines
yes, they're doing same thing to bridge the software nodes needed by v4l2, but they have different type
devices(pci and mei_client device)and dependency. And they have same SSDB definition in DSDT, so the structures are almost same.
Quoted 5 lines
Quoted 2 lines
If there are differences, what are they?
The properties of swnode are same, because we need same info to configure CSI2. But csi bridge can get sensor connected to IVSC by dependency info instead of just polling the only supported sensors for ipu. Also IVSC
It's totally fine the method for finding the sensor is different, this should be an extra argument for the bridge init function and a small number of lines of additional code.
Quoted 1 line
doesn't need vcm. And the topology of dependency is different.
The VCM is optional for current cio2_bridge, too.
It also seems that this currently creates swnodes on mei csi side only for the connection towards the sensor, not the IPU. At the moment you can't reliably add software nodes to a device that has may be already probing so both of these should be created at the same time.
Quoted 5 lines
Quoted 5 lines
What comes to
cio2_bridge, the fact that it's related to a PCI device doesn't seem to matter after initialisation so it could as well work with struct device.Quoted 1 line
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
I have no idea what the ipu bridge would be like, but IVSC csi bridge can be configured via kconfig to enable/disable.
Please work out the details with Bingbu.
@[email protected] @Sakari Ailus Please share your design idea here. What the ipu bridge would be like? What's the responsibility of ipu bridge?
I'd expect Bingbu to have patches to turn the current cio2_bridge to an ipu_bridge at some point. These should come on top of those patches.
Signature · 3 lines
Kind regards, Sakari Ailus
Quoted 112 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Monday, March 27, 2023 5:37 PMHi Wentong,
Quoted 49 lines · Wentong
Quoted 49 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Monday, March 27, 2023 3:50 PMHi Wentong,
Quoted 25 lines · Wentong
Quoted 25 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Monday, March 27, 2023 3:21 PMHi Wentong,
Quoted 16 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor
Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking
up systemQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the
CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces exportedvia v4l2 sub-device.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
modules are:
Quoted 4 lines
- ACE (Algorithm Context Engine): This module is for
algorithm computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership ofcamera sensor.
Quoted 2 lines
- CSI (Camera Serial Interface): This module is used to route
camera sensor data either to IVSC or to host for IPU driver andapplication.
Quoted 34 lines
Quoted 34 lines
Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, users are informed via
v4l2 control API.In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
Implementation:
There are two different drivers to handle ACE and CSI hardware modules inside IVSC.
-ivsc_csi: MEI client driver to send commands and receive
notifications from CSI module.
-ivsc_ace: MEI client driver to send commands and get status
from ACE module.
Interface is exposed via v4l2 sub-devcie APIs to acquire and release camera sensor and CSI-2 link.Quoted 6 lines
Quoted 6 lines
Thanks for the update.
Could you elaborate the decision of keeping the
csi_bridgeentirely separate from thecio2_bridge(to be turned toipu_bridgefirst)? Both are doing essentially the same and using the same data structures,aren't they?
Quoted 5 lines
yes, they're doing same thing to bridge the software nodes needed by v4l2, but they have different type
devices(pci and mei_client device)and dependency. And they have same SSDB definition in DSDT, so the structures are almost same.Quoted 6 lines
Quoted 2 lines
If there are differences, what are they?
The properties of swnode are same, because we need same info to configure CSI2. But csi bridge can get sensor connected to IVSC by dependency info instead of just polling the only supported sensors for ipu. Also IVSC
It's totally fine the method for finding the sensor is different, this should be an extra argument for the bridge init function and a small number of lines of additional code.
Quoted 6 lines
Quoted 1 line
doesn't need vcm. And the topology of dependency is different.
The VCM is optional for current
cio2_bridge, too.It also seems that this currently creates swnodes on mei csi side only for the connection towards the sensor, not the IPU.
IPU could find the MEI CSI software node by software_node_find_by_name, and then setup the bridge between MEI CSI and IPU.
Quoted 3 lines
At the moment you can't reliably
add software nodes to a device that has may be already probing so both of these should be created at the same time.
If you're worried about sequence, IPU driver could make software node and register it, MEI CSI bridge could find IPU software node by software_node_find_by_name, if the return value is NULL, driver will defer the probe.
Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
What comes to
cio2_bridge, the fact that it's related to a PCI device doesn't seem to matter after initialisation so it could as well workwith struct device.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
I have no idea what the ipu bridge would be like, but IVSC csi bridge can be configured via kconfig to enable/disable.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Please work out the details with Bingbu.
@[email protected] @Sakari Ailus Please share your design idea
here.
Quoted 1 line
What the ipu bridge would be like? What's the responsibility of ipu bridge?
I'd expect Bingbu to have patches to turn the current
cio2_bridgeto anipu_bridgeat some point. These should come on top of those patches.
When will this be ready if you already make the plan? But could you please help review my other patches except bridge?
Thanks
Wentong
Quoted 5 lines
--
Kind regards,Sakari Ailus
Sakari Ailus
maintainerQuoted 122 lines · Wentong
Quoted 112 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Monday, March 27, 2023 5:37 PMHi Wentong,
Quoted 49 lines · Wentong
Quoted 49 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Monday, March 27, 2023 3:50 PMHi Wentong,
Quoted 25 lines · Wentong
Quoted 25 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Monday, March 27, 2023 3:21 PMHi Wentong,
Quoted 16 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor
Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking
up systemQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Since the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the
CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces exportedvia v4l2 sub-device.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management Engine) commands. These two hardware modules have two different MEI UUIDs to enumerate. These hardware
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
modules are:
Quoted 4 lines
- ACE (Algorithm Context Engine): This module is for
algorithm computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership ofcamera sensor.
Quoted 2 lines
- CSI (Camera Serial Interface): This module is used to route
camera sensor data either to IVSC or to host for IPU driver andapplication.
Quoted 34 lines
Quoted 34 lines
Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, users are informed via
v4l2 control API.In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
Implementation:
There are two different drivers to handle ACE and CSI hardware modules inside IVSC.
-ivsc_csi: MEI client driver to send commands and receive
notifications from CSI module.
-ivsc_ace: MEI client driver to send commands and get status
from ACE module.
Interface is exposed via v4l2 sub-devcie APIs to acquire and release camera sensor and CSI-2 link.Quoted 6 lines
Quoted 6 lines
Thanks for the update.
Could you elaborate the decision of keeping the
csi_bridgeentirely separate from thecio2_bridge(to be turned toipu_bridgefirst)? Both are doing essentially the same and using the same data structures,aren't they?
Quoted 5 lines
yes, they're doing same thing to bridge the software nodes needed by v4l2, but they have different type
devices(pci and mei_client device)and dependency. And they have same SSDB definition in DSDT, so the structures are almost same.Quoted 6 lines
Quoted 2 lines
If there are differences, what are they?
The properties of swnode are same, because we need same info to configure CSI2. But csi bridge can get sensor connected to IVSC by dependency info instead of just polling the only supported sensors for ipu. Also IVSC
It's totally fine the method for finding the sensor is different, this should be an extra argument for the bridge init function and a small number of lines of additional code.
Quoted 6 lines
Quoted 1 line
doesn't need vcm. And the topology of dependency is different.
The VCM is optional for current
cio2_bridge, too.It also seems that this currently creates swnodes on mei csi side only for the connection towards the sensor, not the IPU.
IPU could find the MEI CSI software node by
software_node_find_by_name, and then setup the bridge between MEI CSI and IPU.
Quoted 7 lines
Quoted 3 lines
At the moment you can't reliably
add software nodes to a device that has may be already probing so both of these should be created at the same time.If you're worried about sequence, IPU driver could make software node and register it, MEI CSI bridge could find IPU software node by
software_node_find_by_name, if the return value is NULL, driver will defer the probe.
The IPU driver would also need to figure out whether a given sensor should be behind the IVSC. I imagine it'll be hard to achieve this reasonably neatly if you split the implementation into different parts --- both of which need the same information from the same place (and in neither case it's under the same device than to which the driver is bound).
Quoted 14 lines
Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
What comes to
cio2_bridge, the fact that it's related to a PCI device doesn't seem to matter after initialisation so it could as well workwith struct device.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
I have no idea what the ipu bridge would be like, but IVSC csi bridge can be configured via kconfig to enable/disable.
Please work out the details with Bingbu.
@[email protected] @Sakari Ailus Please share your design idea
Quoted 8 lines
Quoted 5 lines
here.
Quoted 1 line
What the ipu bridge would be like? What's the responsibility of ipu bridge?
I'd expect Bingbu to have patches to turn the current
cio2_bridgeto anipu_bridgeat some point. These should come on top of those patches.When will this be ready if you already make the plan? But could you please help review my other patches except bridge?
Please configure your e-mail client to wrap your lines at most at 74 characters or so.
I think it's binding this all together that is the concern, the individual drivers much less so.
Signature · 3 lines
Regards, Sakari Ailus
Hi Sakari,
Thanks
Quoted 169 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Wednesday, March 29, 2023 3:49 AMQuoted 51 lines · Wentong
Quoted 51 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Monday, March 27, 2023 5:37 PMHi Wentong,
Quoted 42 lines · Wentong
Quoted 42 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Monday, March 27, 2023 3:50 PMHi Wentong,
Quoted 33 lines · Wentong
Quoted 33 lines
-----Original Message-----
From: Sakari Ailus <[email protected]>
Sent: Monday, March 27, 2023 3:21 PMHi Wentong,
Quoted 24 lines · Wentong Wu
Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a companion chip designed to provide secure and low power vision capability to IA platforms. IVSC is available in existing commercial platforms from multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC interfaces directly with the platform main camera sensor via a CSI-2 link and processes the image data with the embedded AI engine. The detected events are sent over I2C to ISH (Intel Sensor
Hub) for additional data fusion from multiple sensors. The fusion results are used to implement advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or
waking up systemSince the Image Processing Unit(IPU) used on the host processor needs to configure the CSI-2 link in normal camera usages, the
CSI-2 link and camera sensor can only be used in mutually-exclusive ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU driver can take ownership of the CSI-2 link and camera sensor using interfaces exportedQuoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
via v4l2 sub-device.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Switching ownership requires an interface with two different hardware modules inside IVSC. The software interface to these modules is via Intel MEI (The Intel Management
Engine) commands.
Quoted 33 lines
Quoted 33 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines
These two hardware modules have two different MEI UUIDs to enumerate. These hardware
modules are:
Quoted 4 lines
- ACE (Algorithm Context Engine): This module is for
algorithm computing when IVSC owns camera sensor. Also ACE module controls camera sensor's ownership. This hardware module is used to set ownership ofcamera sensor.
Quoted 3 lines
- CSI (Camera Serial Interface): This module is used to
route camera sensor data either to IVSC or to host for IPU driver andapplication.
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
IVSC also provides a privacy mode. When privacy mode is turned on, camera sensor can't be used. This means that both ACE and host IPU can't get image data. And when this mode is turned on, users are informed via
v4l2 control API.In summary, to acquire ownership of camera by IPU driver, first ACE module needs to be informed of ownership and then to setup MIPI CSI-2 link for the camera sensor and IPU.
Implementation:
There are two different drivers to handle ACE and CSI hardware modules inside IVSC.
-ivsc_csi: MEI client driver to send commands and
receive notifications from CSI module.
-ivsc_ace: MEI client driver to send commands and get
status from ACE module.
Interface is exposed via v4l2 sub-devcie APIs to acquire and release camera sensor and CSI-2 link.Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Thanks for the update.
Could you elaborate the decision of keeping the
csi_bridgeentirely separate from thecio2_bridge(to be turned toipu_bridgefirst)?
Quoted 23 lines
Quoted 23 lines
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines
Both are doing essentially the same and using the same data structures,
aren't they?
Quoted 6 lines
yes, they're doing same thing to bridge the software nodes needed by v4l2, but they have different type devices(pci and
mei_client
device) and dependency. And they have same SSDB definition in DSDT, so the structures are almost same.Quoted 6 lines
Quoted 2 lines
If there are differences, what are they?
The properties of swnode are same, because we need same info to configure CSI2. But csi bridge can get sensor connected to IVSC by dependency info instead of just polling the only supported sensors for ipu. Also IVSC
It's totally fine the method for finding the sensor is different, this should be an extra argument for the bridge init function and a small number of lines of additional code.
Quoted 1 line
doesn't need vcm. And the topology of dependency is different.
Quoted 7 lines
Quoted 4 lines
The VCM is optional for current
cio2_bridge, too.It also seems that this currently creates swnodes on mei csi side only for the connection towards the sensor, not the IPU.
IPU could find the MEI CSI software node by
software_node_find_by_name, and then setup the bridge between MEI CSIand IPU.
Quoted 8 lines
Quoted 4 lines
At the moment you can't reliably
add software nodes to a device that has may be already probing so both of these should be created at the same time.If you're worried about sequence, IPU driver could make software node and register it, MEI CSI bridge could find IPU software node by
software_node_find_by_name, if the return value is NULL, driver will defer theprobe.
The IPU driver would also need to figure out whether a given sensor should be behind the IVSC. I imagine it'll be hard to achieve this reasonably neatly if you split the implementation into different parts --- both of which need the same information from the same place (and in neither case it's under the same device than to which the driver is bound).
Quoted 22 lines
Quoted 21 lines
Quoted 6 lines
Quoted 6 lines
What comes to
cio2_bridge, the fact that it's related to a PCI device doesn't seem to matter after initialisation so it could as well workwith struct device.
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
I have no idea what the ipu bridge would be like, but IVSC csi bridge can be configured via kconfig to enable/disable.
Please work out the details with Bingbu.
@[email protected] @Sakari Ailus Please share your design idea
here.
Quoted 1 line
What the ipu bridge would be like? What's the responsibility of ipu bridge?
I'd expect Bingbu to have patches to turn the current
cio2_bridgeto anipu_bridgeat some point. These should come on top of those patches.
Quoted 8 lines
Quoted 2 lines
When will this be ready if you already make the plan? But could you please help review my other patches except bridge?
Please configure your e-mail client to wrap your lines at most at 74 characters or so.
I think it's binding this all together that is the concern, the individual drivers much less so.
Thanks, I would remove the csi bridge in next version.
Ok, the IPU bridge will also cover the bridge between IVSC and sensor.
BTW, please hans and laurent help take a look the mei_csi and mei_ace patch.
BR,
Wentong
Quoted 5 lines
--
Regards,Sakari Ailus
Sakari Ailus
maintainerHi Wentong,
Quoted 22 lines · Wentong
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
I have no idea what the ipu bridge would be like, but IVSC csi bridge can be configured via kconfig to enable/disable.
Please work out the details with Bingbu.
@[email protected] @Sakari Ailus Please share your design idea
here.
Quoted 1 line
What the ipu bridge would be like? What's the responsibility of ipu bridge?
I'd expect Bingbu to have patches to turn the current
cio2_bridgeto anipu_bridgeat some point. These should come on top of those patches.Quoted 8 lines
Quoted 2 lines
When will this be ready if you already make the plan? But could you please help review my other patches except bridge?
Please configure your e-mail client to wrap your lines at most at 74 characters or so.
I think it's binding this all together that is the concern, the individual drivers much less so.
Quoted 6 lines
Thanks, I would remove the csi bridge in next version.
Ok, the IPU bridge will also cover the bridge between IVSC and sensor.
BTW, please hans and laurent help take a look the
mei_csiandmei_acepatch.
I can review them. However they can't be tested without the bridge code in place. I'm therefore leaning towards merging them all at the same time, when we know how this looks like in the end and can actually test the patches.
Signature · 3 lines
Regards, Sakari Ailus