media: platform: Add support for Face Detection (FD) on mt8183 SoC
Subsystems (3)
v3 cover media: platform: Add support for Face Detection (FD) on mt8183 SoC
Hello,
This RFC patch series is adding Face Detection (FD) driver on Mediatek mt8183 SoC. It belongs to the first Mediatek's camera driver series based on V4L2 and media controller framework. I posted the main part of the FD driver as RFC to discuss first and would like some review comments.
==============
Introduction
==============
Face Detection (FD) unit provides hardware accelerated face detection feature. It can detect different sizes of faces in a given image.
The driver is implemented as a normal V4L2 memory-to-memory device and supports V4L2 controls for detection settings. It has two buffer queues.
1. Video output buffer: RAW image for face detection.
2. Meta capture buffer: Result of the detected faces.
==================
Changes in v3
==================
RFC v3 includes the following modification:
1. Adjust the private control ids and place them in include/uapi/linux/mtk-fd-v4l2-controls.h
2. Fixing review commnets in mtk_fd_40.c and mtk_fd.h
3. Define FD meta capture buffer in include/uapi/linux/videodev2.h
4. Remove the usage of get_reserved_memory by scp driver and use dma_alloc api instead
Todo:
- Add v4l2 control menus for private mtk fd control
- Refine the job finish flow when system suspend
==================
Changes in v2
==================
RFC v2 includes the following modification:
1. Implement FD as a V4L2 mem2mem driver
2. Replace meta input with V4L2 controls
==================
Changes in v1
==================
RFC v1 includes the following modification:
1. Uses Request API instead of FD's buffer collection design
2. removed unnecessary abstraction structurally, including mtk_fd_ctx and related ops
3. removed the fd_smem node from device tree
4. Fixed the common issues Tomasz commented on Mediatek ISP Pass 1's RFC v0 patch series
==================
Dependent patch
==================
FD driver depends on SCP driver. The patches are as following:
[1]. Add support for mt8183 SCP
https://patchwork.kernel.org/cover/11132177/
==================
Compliance test
==================
* Version: https://git.linuxtv.org/v4l-utils.git/commit/?id=b16f9e945d74aa5
* Note: Some of the failures are caused by the implementation of FD driver, which is a m2m device with VIDEO_OUT and META_CAPTURE queues, therefore we can't set V4L2_CAP_VIDEO_M2M in device capability, and fail in some non-m2m v4l2 test cases.
* Test command: v4l2-compliance -m 0
* test output:
v4l2-compliance SHA: not available, 32 bits
Compliance test for mtk-fd-4.0 device /dev/media0:
Media Driver Info:
Driver name : mtk-fd-4.0
Model : mtk-fd-4.0
Serial :
Bus info : platform:1502b000.fd
Media version : 4.19.67
Hardware revision: 0x00000000 (0)
Driver version : 4.19.67Required ioctls:
test MEDIA_IOC_DEVICE_INFO: OK
Allow for multiple opens:
test second /dev/media0 open: OK
test MEDIA_IOC_DEVICE_INFO: OK
test for unlimited opens: OK
Media Controller ioctls:
test MEDIA_IOC_G_TOPOLOGY: OK
Entities: 3 Interfaces: 1 Pads: 4 Links: 4 test MEDIA_IOC_ENUM_ENTITIES/LINKS: OK test MEDIA_IOC_SETUP_LINK: OK
Total for mtk-fd-4.0 device /dev/media0: 7, Succeeded: 7, Failed: 0, Warnings: 0
--------------------------------------------------------------------------------
Compliance test for mtk-fd-4.0 device /dev/video0:
Driver Info:
Driver name : mtk-fd-4.0
Card type : mtk-fd-4.0
Bus info : platform:1502b000.fd
Driver version : 4.19.67
Capabilities : 0x84a02000 Video Output Multiplanar
Metadata Capture
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04a02000
Video Output Multiplanar
Metadata Capture
Streaming
Extended Pix Format
Media Driver Info:
Driver name : mtk-fd-4.0
Model : mtk-fd-4.0
Serial :
Bus info : platform:1502b000.fd
Media version : 4.19.67
Hardware revision: 0x00000000 (0)
Driver version : 4.19.67Interface Info:
ID : 0x0300000c
Type : V4L Video
Entity Info:
ID : 0x00000001 (1)
Name : mtk-fd-4.0-source
Function : V4L2 I/O
Pad 0x01000002 : 0: Source
Link 0x02000008: to remote pad 0x1000005 of entity 'mtk-fd-4.0-proc': Data, Enabled, ImmutableRequired ioctls:
test MC information (see 'Media Driver Info' above): OK
fail: v4l2-compliance.cpp(668): dcaps & output_caps
test VIDIOC_QUERYCAP: FAIL
Allow for multiple opens:
test second /dev/video0 open: OK
fail: v4l2-compliance.cpp(668): dcaps & output_caps
test VIDIOC_QUERYCAP: FAIL
test VIDIOC_G/S_PRIORITY: OK
test for unlimited opens: OK
Debug ioctls:
test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported) test VIDIOC_LOG_STATUS: OK (Not Supported)
Input ioctls:
test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported) test VIDIOC_ENUMAUDIO: OK (Not Supported)
test VIDIOC_G/S/ENUMINPUT: OK (Not Supported) test VIDIOC_G/S_AUDIO: OK (Not Supported)
Inputs: 0 Audio Inputs: 0 Tuners: 0
Output ioctls:
test VIDIOC_G/S_MODULATOR: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_ENUMAUDOUT: OK (Not Supported) test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported) test VIDIOC_G/S_AUDOUT: OK (Not Supported)
Outputs: 0 Audio Outputs: 0 Modulators: 0
Input/Output configuration ioctls:
test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported) test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported) test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported) test VIDIOC_G/S_EDID: OK (Not Supported)
Control ioctls:
test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK test VIDIOC_QUERYCTRL: OK
test VIDIOC_G/S_CTRL: OK
test VIDIOC_G/S/TRY_EXT_CTRLS: OK test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK test VIDIOC_G/S_JPEGCOMP: OK (Not Supported) Standard Controls: 1 Private Controls: 6
Format ioctls:
test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
test VIDIOC_G/S_PARM: OK (Not Supported)
test VIDIOC_G_FBUF: OK (Not Supported)
fail: v4l2-test-formats.cpp(457): pixelformat 00000000 () for buftype 10 not reported by ENUM_FMT
test VIDIOC_G_FMT: FAIL
fail: v4l2-test-formats.cpp(457): pixelformat 00000000 () for buftype 10 not reported by ENUM_FMT
test VIDIOC_TRY_FMT: FAIL
fail: v4l2-test-formats.cpp(457): pixelformat ffffffff (-BE) for buftype 10 not reported by ENUM_FMT
test VIDIOC_S_FMT: FAIL
test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
test Cropping: OK (Not Supported)
test Composing: OK (Not Supported)
test Scaling: OK (Not Supported)
Codec ioctls:
test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported) test VIDIOC_G_ENC_INDEX: OK (Not Supported) test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
Buffer ioctls:
fail: v4l2-test-buffers.cpp(667): q2.reqbufs(node->node2, 1) != EBUSY
test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: FAIL
test VIDIOC_EXPBUF: OK
test Requests: OK
Total for mtk-fd-4.0 device /dev/video0: 45, Succeeded: 39, Failed: 6, Warnings: 0
Grand Total for mtk-fd-4.0 device /dev/media0: 52, Succeeded: 46, Failed: 6, Warnings: 0
RFC PATCH v3 1/3 dt-bindings: mt8183: Added FD dt-bindings
From: Jerry-ch Chen <[email protected]>
This patch adds DT binding documentation for the Face Detection (FD) unit of the Mediatek's mt8183 SoC.
--- .../bindings/media/mediatek,mt8183-fd.txt | 34 +++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100644 Documentation/devicetree/bindings/media/mediatek,mt8183-fd.txt
Documentation/devicetree/bindings/media/mediatek,mt8183-fd.txt new file +34 -0
@@ -0,0 +1,34 @@ +* Mediatek Face Detection Unit (FD) + +Face Detection (FD) unit is a typical memory-to-memory HW device. +It provides hardware accelerated face detection function, and it +is able to detect different poses of faces. FD will writre result +of detected face into memory as output. + +Required properties: +- compatible: "mediatek,mt8183-fd" +- mediatek,scp : the node of system control processor (SCP), see + Documentation/devicetree/bindings/remoteproc/mtk,scp.txt for details. +- iommus: should point to the 3 entries: M4U_PORT_CAM_FDVT_RP, + M4U_PORT_CAM_FDVT_WR and M4U_PORT_CAM_FDVT_RB. (Please see + Documentation/devicetree/bindings/iommu/mediatek,iommu.txt for details.) +- reg: Physical base address and length of the register space. +- interrupts: interrupt number to the cpu. +- clocks : must contain the FDVT clock, see + Documentation/devicetree/bindings/clock/clock-bindings.txt for details. +- clock-names: must be "fd". +- power-domain: must be "MT8183_POWER_DOMAIN_ISP". + +Example: + fd: fd@1502b000 { + compatible = "mediatek,mt8183-fd"; + mediatek,scp = <&scp>; + iommus = <&iommu M4U_PORT_CAM_FDVT_RP>, + <&iommu M4U_PORT_CAM_FDVT_WR>, + <&iommu M4U_PORT_CAM_FDVT_RB>; + reg = <0 0x1502b000 0 0x1000>; + interrupts = <GIC_SPI 269 IRQ_TYPE_LEVEL_LOW>; + clocks = <&imgsys CLK_IMG_FDVT>; + clock-names = "fd"; + power-domains = <&scpsys MT8183_POWER_DOMAIN_ISP>; + };
Signature · 1 line
2.18.0
Re: [RFC PATCH V3 1/3] dt-bindings: mt8183: Added FD dt-bindings
Quoted 11 lines · Jerry-ch Chen
From: Jerry-ch Chen <jerry-ch.chen-NuS5LvNUpcJWk0Htik3J/[email protected]>
This patch adds DT binding documentation for the Face Detection (FD) unit of the Mediatek's mt8183 SoC.
--- .../bindings/media/mediatek,mt8183-fd.txt | 34 +++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100644 Documentation/devicetree/bindings/media/mediatek,mt8183-fd.txt
RFC PATCH v3 2/3 dts: arm64: mt8183: Add FD nodes
In reply to Jerry-ch ChenFrom: Jerry-ch Chen <[email protected]>
This patch adds nodes for Face Detection (FD) unit. FD is embedded in Mediatek SoCs and works with the co-processor to perform face detection on the input data and image and output detected face result.
--- arch/arm64/boot/dts/mediatek/mt8183.dtsi | 13 +++++++++++++ 1 file changed, 13 insertions(+)
arch/arm64/boot/dts/mediatek/mt8183.dtsi +13 -0
@@ -432,6 +432,19 @@ #clock-cells = <1>; }; + fd:fd@1502b000 { + compatible = "mediatek,mt8183-fd"; + mediatek,scp = <&scp>; + iommus = <&iommu M4U_PORT_CAM_FDVT_RP>, + <&iommu M4U_PORT_CAM_FDVT_WR>, + <&iommu M4U_PORT_CAM_FDVT_RB>; + reg = <0 0x1502b000 0 0x1000>; + interrupts = <GIC_SPI 269 IRQ_TYPE_LEVEL_LOW>; + clocks = <&imgsys CLK_IMG_FDVT>; + clock-names = "fd"; + power-domains = <&scpsys MT8183_POWER_DOMAIN_ISP>; + }; + vdecsys: syscon@16000000 { compatible = "mediatek,mt8183-vdecsys", "syscon"; reg = <0 0x16000000 0 0x1000>;
Signature · 1 line
2.18.0
RFC PATCH v3 3/3 platform: mtk-isp: Add Mediatek FD driver
In reply to Jerry-ch ChenFrom: Jerry-ch Chen <[email protected]>
This patch adds the driver of Face Detection (FD) unit in Mediatek camera system, providing face detection function.
The mtk-isp directory will contain drivers for multiple IP blocks found in Mediatek ISP system. It will include ISP Pass 1 driver (CAM), sensor interface driver, DIP driver and face detection driver.
--- drivers/media/platform/Kconfig | 2 + drivers/media/platform/Makefile | 2 + drivers/media/platform/mtk-isp/fd/Kconfig | 19 + drivers/media/platform/mtk-isp/fd/Makefile | 5 + drivers/media/platform/mtk-isp/fd/mtk_fd.h | 148 ++ drivers/media/platform/mtk-isp/fd/mtk_fd_40.c | 1219 +++++++++++++++++ include/uapi/linux/mtk-fd-v4l2-controls.h | 69 + include/uapi/linux/v4l2-controls.h | 4 + 8 files changed, 1468 insertions(+) create mode 100644 drivers/media/platform/mtk-isp/fd/Kconfig create mode 100644 drivers/media/platform/mtk-isp/fd/Makefile create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd.h create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd_40.c create mode 100644 include/uapi/linux/mtk-fd-v4l2-controls.h
drivers/media/platform/Kconfig +2 -0
@@ -32,6 +32,8 @@ source "drivers/media/platform/davinci/Kconfig" source "drivers/media/platform/omap/Kconfig" +source "drivers/media/platform/mtk-isp/fd/Kconfig" + config VIDEO_ASPEED tristate "Aspeed AST2400 and AST2500 Video Engine driver" depends on VIDEO_V4L2
drivers/media/platform/Makefile +2 -0
@@ -94,6 +94,8 @@ obj-$(CONFIG_VIDEO_MEDIATEK_MDP) += mtk-mdp/ obj-$(CONFIG_VIDEO_MEDIATEK_JPEG) += mtk-jpeg/ +obj-$(CONFIG_VIDEO_MEDIATEK_FD) += mtk-isp/fd/ + obj-$(CONFIG_VIDEO_QCOM_CAMSS) += qcom/camss/ obj-$(CONFIG_VIDEO_QCOM_VENUS) += qcom/venus/
drivers/media/platform/mtk-isp/fd/Kconfig new file +19 -0
@@ -0,0 +1,19 @@ +config VIDEO_MEDIATEK_FD + bool "Mediatek face detection processing function" + depends on VIDEO_V4L2 + depends on ARCH_MEDIATEK || COMPILE_TEST + select VIDEOBUF2_DMA_CONTIG + select VIDEOBUF2_CORE + select VIDEOBUF2_V4L2 + select VIDEOBUF2_MEMOPS + select MEDIA_CONTROLLER + select MTK_SCP + + default n + help + Support the Face Detection (FD) feature in the Mediatek + mt8183 Soc. + + FD driver is a V4L2 memory-to-memory device driver which + provides hardware accelerated face detection function, + it can detect different sizes of faces in a raw image.
drivers/media/platform/mtk-isp/fd/Makefile new file +5 -0
@@ -0,0 +1,5 @@ +# SPDX-License-Identifier: GPL-2.0 + +mtk-fd-objs += mtk_fd_40.o + +obj-$(CONFIG_VIDEO_MEDIATEK_FD) += mtk-fd.o
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +148 -0
148 lines not on this page · raw message
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +1219 -0
1,219 lines not on this page · raw message
include/uapi/linux/mtk-fd-v4l2-controls.h new file +69 -0
69 lines not on this page · raw message
include/uapi/linux/v4l2-controls.h +4 -0
@@ -192,6 +192,10 @@ enum v4l2_colorfx {* We reserve 16 controls for this driver. */ #define V4L2_CID_USER_IMX_BASE (V4L2_CID_USER_BASE + 0x10b0) +/* The base for the mediatek FD driver controls */ +/* We reserve 16 controls for this driver. */ +#define V4L2_CID_USER_MTK_FD_BASE (V4L2_CID_USER_BASE + 0x10d0) +/* MPEG-class control IDs */ /* The MPEG controls are applicable to all codec controls * and the 'MPEG' part of the define is historical */
Signature · 1 line
2.18.0
Re: [RFC PATCH V3 3/3] platform: mtk-isp: Add Mediatek FD driver
Hi Tomasz,
Quoted 27 lines · Jerry-ch Chen
From: Jerry-ch Chen <[email protected]>
This patch adds the driver of Face Detection (FD) unit in Mediatek camera system, providing face detection function.
The mtk-isp directory will contain drivers for multiple IP blocks found in Mediatek ISP system. It will include ISP Pass 1 driver (CAM), sensor interface driver, DIP driver and face detection driver.
Signed-off-by Jerry-ch Chen <[email protected]>--- drivers/media/platform/Kconfig | 2 + drivers/media/platform/Makefile | 2 + drivers/media/platform/mtk-isp/fd/Kconfig | 19 + drivers/media/platform/mtk-isp/fd/Makefile | 5 + drivers/media/platform/mtk-isp/fd/mtk_fd.h | 148 ++ drivers/media/platform/mtk-isp/fd/mtk_fd_40.c | 1219 +++++++++++++++++ include/uapi/linux/mtk-fd-v4l2-controls.h | 69 + include/uapi/linux/v4l2-controls.h | 4 + 8 files changed, 1468 insertions(+) create mode 100644 drivers/media/platform/mtk-isp/fd/Kconfig create mode 100644 drivers/media/platform/mtk-isp/fd/Makefile create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd.h create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd_40.c create mode 100644 include/uapi/linux/mtk-fd-v4l2-controls.h
[snip]
Quoted 231 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +229
-0+ +static void mtk_fd_hw_job_finish (struct mtk_fd_dev *fd, +enum vb2_buffer_state vb_state) +{ +struct mtk_fd_ctx *ctx; +struct vb2_v4l2_buffer *src_vbuf =NULL , *dst_vbuf =NULL ; + + ctx =v4l2_m2m_get_curr_priv (fd->m2m_dev); + src_vbuf =v4l2_m2m_src_buf_remove (ctx->fh.m2m_ctx); + dst_vbuf =v4l2_m2m_dst_buf_remove (ctx->fh.m2m_ctx); + +v4l2_m2m_buf_copy_metadata (src_vbuf, dst_vbuf, +V4L2_BUF_FLAG_TSTAMP_SRC_MASK ); +v4l2_m2m_buf_done (src_vbuf, vb_state); +v4l2_m2m_buf_done (dst_vbuf, vb_state); +v4l2_m2m_job_finish (fd->m2m_dev, ctx->fh.m2m_ctx); +} + +static void mtk_fd_ipi_handler (void *data,unsigned int len,void *priv) +{ +struct mtk_fd_dev *fd = (struct mtk_fd_dev *)priv; +struct device *dev = fd->dev; +struct ipi_message *fd_ack_msg = (struct ipi_message *)data; +struct fd_ack_param *fd_ack = &fd_ack_msg->fd_ack_param; + +dev_dbg (fd->dev,"fd_ipi_ack_id: %d\n" , fd_ack_msg->cmd_id); +switch (fd_ack_msg->cmd_id) { +case MTK_FD_IPI_CMD_INIT_ACK : +return ; +case MTK_FD_IPI_CMD_ENQ_ACK : +if (fd_ack->ret_code) { +dev_err (dev,"ipi ret: %d, message: %d\n" , + fd_ack->ret_code, fd_ack->ret_msg); +pm_runtime_put ((fd->dev)); +mtk_fd_hw_job_finish (fd,VB2_BUF_STATE_ERROR ); + } +return ; +case MTK_FD_IPI_CMD_RESET_ACK : +return ; + } +} + +static int mtk_fd_hw_connect (struct mtk_fd_dev *fd) +{ +int ret; + + ret =rproc_boot (fd->rproc_handle); + +if (ret <0 ) { +/** +* Return 0 if downloading firmware successfully, +* otherwise it is failed +*/ +dev_err (fd->dev,"Failed to boot rproc\n" ); +return ret; + } + + ret =scp_ipi_register (fd->scp_pdev,SCP_IPI_FD_CMD , + mtk_fd_ipi_handler, fd); +if (ret) { +dev_err (fd->dev,"Failed to register IPI cmd handler\n" ); + goto err_rproc_shutdown; + } + + fd->fd_stream_count++; +if (fd->fd_stream_count ==1 ) { +if (mtk_fd_hw_enable (fd)) { + ret = -EINVAL ; + goto err_rproc_shutdown; + } + } +return 0 ; + +err_rproc_shutdown: +rproc_shutdown (fd->rproc_handle); +return ret; +} + +static void mtk_fd_hw_disconnect (struct mtk_fd_dev *fd) +{ + fd->fd_stream_count--; + +if (fd->fd_stream_count ==0 ) { +if (!IS_ERR (fd->rs_dma_buf)) +mtk_fd_free_dma_handle (fd); + +rproc_shutdown (fd->rproc_handle); + } +} + +static int mtk_fd_hw_job_exec (struct mtk_fd_dev *fd, +struct fd_enq_param *fd_param) +{ +struct ipi_message fd_ipi_msg; +int ret; + +pm_runtime_get_sync ((fd->dev)); + +reinit_completion (&fd->fd_irq_done); + fd_ipi_msg.cmd_id =MTK_FD_IPI_CMD_ENQUEUE ; +memcpy (&fd_ipi_msg.fd_enq_param, fd_param,sizeof (struct fd_enq_param )); + ret =scp_ipi_send (fd->scp_pdev,SCP_IPI_FD_CMD , &fd_ipi_msg, +sizeof (fd_ipi_msg),MTK_FD_IPI_SEND_TIMEOUT ); +if (ret) { +pm_runtime_put ((fd->dev)); +mtk_fd_hw_job_finish (fd,VB2_BUF_STATE_ERROR ); +return ret; + } +return 0 ; +} + +static int mtk_fd_vb2_buf_out_validate (struct vb2_buffer *vb) +{ +struct vb2_v4l2_buffer *v4l2_buf =to_vb2_v4l2_buffer (vb); + +if (v4l2_buf->field ==V4L2_FIELD_ANY ) + v4l2_buf->field =V4L2_FIELD_NONE ; +if (v4l2_buf->field !=V4L2_FIELD_NONE ) +return -EINVAL ; + +return 0 ; +} + +static int mtk_fd_vb2_buf_prepare (struct vb2_buffer *vb) +{ +struct vb2_v4l2_buffer *vbuf =to_vb2_v4l2_buffer (vb); +struct vb2_queue *vq = vb->vb2_queue; +struct mtk_fd_ctx *ctx =vb2_get_drv_priv (vq); +struct device *dev = ctx->dev; +struct v4l2_pix_format_mplane *pixfmt; + +switch (vq->type) { +case V4L2_BUF_TYPE_META_CAPTURE : +if (vb2_plane_size (vb,0 ) < ctx->dst_fmt.buffersize) { +dev_dbg (dev,"meta size %d is too small\n" ); +return -EINVAL ; + } +break ; +case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE : + pixfmt = &ctx->src_fmt; + +if (vbuf->field ==V4L2_FIELD_ANY ) + vbuf->field =V4L2_FIELD_NONE ; + +if (vb->num_planes >2 || vbuf->field !=V4L2_FIELD_NONE ) { +dev_dbg (dev,"plane or field %d not supported\n" , + vb->num_planes, vbuf->field); +return -EINVAL ; + } +if (vb2_plane_size (vb,0 ) < pixfmt->plane_fmt[0 ].sizeimage) { +dev_dbg (dev,"plane %d is too small\n" ); +return -EINVAL ; + } +break ; + } + +return 0 ; +} + +static void mtk_fd_vb2_buf_queue (struct vb2_buffer *vb) +{ +struct mtk_fd_ctx *ctx =vb2_get_drv_priv (vb->vb2_queue); +struct vb2_v4l2_buffer *vbuf =to_vb2_v4l2_buffer (vb); + +v4l2_m2m_buf_queue (ctx->fh.m2m_ctx, vbuf); +} + +static int mtk_fd_vb2_queue_setup (struct vb2_queue *vq, +unsigned int *num_buffers, +unsigned int *num_planes, +unsigned int sizes[], +struct device *alloc_devs[]) +{ +struct mtk_fd_ctx *ctx =vb2_get_drv_priv (vq); +unsigned int size[2 ]; +unsigned int plane; + +switch (vq->type) { +case V4L2_BUF_TYPE_META_CAPTURE : + size[0 ] = ctx->dst_fmt.buffersize; +break ; +case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE : + size[0 ] = ctx->src_fmt.plane_fmt[0 ].sizeimage; +if (*num_planes ==2 ) + size[1 ] = ctx->src_fmt.plane_fmt[1 ].sizeimage; +break ; + } + +if (*num_planes >2 ) +return -EINVAL ; +if (*num_planes ==0 ) { +if (vq->type ==V4L2_BUF_TYPE_META_CAPTURE ) { + sizes[0 ] = ctx->dst_fmt.buffersize; + *num_planes =1 ; +return 0 ; + } + + *num_planes = ctx->src_fmt.num_planes; +for (plane =0 ; plane < *num_planes; plane++) + sizes[plane] = ctx->src_fmt.plane_fmt[plane].sizeimage; +return 0 ; + } + +for (plane =0 ; plane < *num_planes; plane++) { +if (sizes[plane] < size[plane]) +return -EINVAL ; + } +return 0 ; +} + +static int mtk_fd_vb2_start_streaming (struct vb2_queue *vq,unsigned int count) +{ +struct mtk_fd_ctx *ctx =vb2_get_drv_priv (vq); + +if (vq->type ==V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE ) +return mtk_fd_hw_connect (ctx->fd_dev); +else +return 0 ; +} + +static int mtk_fd_job_abort (struct mtk_fd_dev *fd) +{ +u32 ret; + + ret =wait_for_completion_timeout (&fd->fd_irq_done, +msecs_to_jiffies (MTK_FD_HW_TIMEOUT )); +/* Reset FD HW */ +if (!ret) { +struct ipi_message fd_ipi_msg;
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c line 362
Quoted 10 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +10
-0+ + fd_ipi_msg.cmd_id =MTK_FD_IPI_CMD_RESET ; +if (scp_ipi_send (fd->scp_pdev,SCP_IPI_FD_CMD , &fd_ipi_msg, +sizeof (fd_ipi_msg),MTK_FD_IPI_SEND_TIMEOUT )) +dev_err (fd->dev,"FD Reset HW error\n" ); +return -ETIMEDOUT ; + } +return 0 ; +} +
Continue the discussion about job abort in RFC v2,
I think the job_abort callback in v4l2_m2m_ops() might be useful.
ref:
https://elixir.bootlin.com/linux/v5.4-rc2/source/drivers/media/v4l2-core/v4l2-mem2mem.c#L398
https://elixir.bootlin.com/linux/v5.4-rc2/source/include/media/v4l2-mem2mem.h#L43
in drivers/media/v4l2-core/v4l2-mem2mem.c #398 v4l2_m2m_cancel_job()
...
if (m2m_ctx->job_flags & TRANS_RUNNING) {
spin_unlock_irqrestore(&m2m_dev->job_spinlock, flags);
if (m2m_dev->m2m_ops->job_abort)
m2m_dev->m2m_ops->job_abort(m2m_ctx->priv);
dprintk("m2m_ctx %p running, will wait to complete\n", m2m_ctx);
wait_event(m2m_ctx->finished,
!(m2m_ctx->job_flags & TRANS_RUNNING));
} ...If this operation is set, we might use the v4l2_m2m_cancel_job() when suspend, and it will do mtk_fd_job_abort() here.
Quoted 19 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +19
-0+static void mtk_fd_vb2_stop_streaming (struct vb2_queue *vq) +{ +struct mtk_fd_ctx *ctx =vb2_get_drv_priv (vq); +struct mtk_fd_dev *fd = ctx->fd_dev; +struct vb2_v4l2_buffer *vb; +struct v4l2_m2m_ctx *m2m_ctx = ctx->fh.m2m_ctx; +struct v4l2_m2m_queue_ctx *queue_ctx; + +mtk_fd_job_abort (fd); + queue_ctx =V4L2_TYPE_IS_OUTPUT (vq->type) ? + &m2m_ctx->out_q_ctx : + &m2m_ctx->cap_q_ctx; +while ((vb =v4l2_m2m_buf_remove (queue_ctx))) +v4l2_m2m_buf_done (vb,VB2_BUF_STATE_ERROR ); + +if (vq->type ==V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE ) +mtk_fd_hw_disconnect (fd); +} +
[snip]
Quoted 60 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +60
-0+struct v4l2_ctrl_config mtk_fd_controls[] = { + { + .id =V4L2_CID_MTK_FD_SCALE_DOWN_IMG_WIDTH , + .name ="FD scale image widths" , + .type =V4L2_CTRL_TYPE_U16 , + .min =MTK_FD_OUTPUT_MIN_WIDTH , + .max =MTK_FD_OUTPUT_MAX_WIDTH , + .step =1 , + .def =MTK_FD_OUTPUT_MAX_WIDTH , + .dims = {MTK_FD_SCALE_ARR_NUM }, + }, + { + .id =V4L2_CID_MTK_FD_SCALE_DOWN_IMG_HEIGHT , + .name ="FD scale image heights" , + .type =V4L2_CTRL_TYPE_U16 , + .min =MTK_FD_OUTPUT_MIN_HEIGHT , + .max =MTK_FD_OUTPUT_MAX_HEIGHT , + .step =1 , + .def =MTK_FD_OUTPUT_MAX_HEIGHT , + .dims = {MTK_FD_SCALE_ARR_NUM }, + }, + { + .id =V4L2_CID_MTK_FD_SCALE_IMG_NUM , + .name ="FD scale size counts" , + .type =V4L2_CTRL_TYPE_INTEGER , + .min =0 , + .max =MTK_FD_SCALE_ARR_NUM , + .step =1 , + .def =0 , + }, + { + .id =V4L2_CID_MTK_FD_DETECT_POSE , + .name ="FD detect face angle and directions" , + .type =V4L2_CTRL_TYPE_U16 , + .min =0 , + .max =0xffff , + .step =1 , + .def =0x3ff , + .dims = {MTK_FD_FACE_ANGLE_NUM }, + }, + { + .id =V4L2_CID_MTK_FD_DETECT_SPEED , + .name ="FD detection speedup" , + .type =V4L2_CTRL_TYPE_INTEGER , + .min =0 , + .max =MTK_FD_MAX_SPEEDUP , + .step =1 , + .def =0 , + }, + { + .id =V4L2_CID_MTK_FD_DETECTION_MODEL , + .name ="FD use extra model" , + .type =V4L2_CTRL_TYPE_BOOLEAN , + .min =0 , + .max =1 , + .step =1 , + .def =0 , + }, +}; +
[snip]
Quoted 96 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +96
-0+static int mtk_vfd_release (struct file *filp) +{ +struct mtk_fd_ctx *ctx =container_of (filp->private_data, + struct mtk_fd_ctx, fh); + +v4l2_m2m_ctx_release (ctx->fh.m2m_ctx); + +v4l2_ctrl_handler_free (&ctx->hdl); +v4l2_fh_del (&ctx->fh); +v4l2_fh_exit (&ctx->fh); + +kfree (ctx); + +return 0 ; +} + +static const struct v4l2_file_operations fd_video_fops = { + .owner =THIS_MODULE , + .open = mtk_vfd_open, + .release = mtk_vfd_release, + .poll = v4l2_m2m_fop_poll, + .unlocked_ioctl = video_ioctl2, + .mmap = v4l2_m2m_fop_mmap, +#ifdef CONFIG_COMPAT + .compat_ioctl32 = v4l2_compat_ioctl32, +#endif + +}; + +static void mtk_fd_fill_user_param (struct user_param *user_param, +struct v4l2_ctrl_handler *hdl) +{ +struct v4l2_ctrl *ctrl; +int i; + + ctrl =v4l2_ctrl_find (hdl,V4L2_CID_MTK_FD_SCALE_DOWN_IMG_WIDTH ); +if (ctrl) +for (i =0 ; i < ctrl->elems; i++) + user_param->scale_img_width[i] = ctrl->p_new.p_u16[i]; + ctrl =v4l2_ctrl_find (hdl,V4L2_CID_MTK_FD_SCALE_DOWN_IMG_HEIGHT ); +if (ctrl) +for (i =0 ; i < ctrl->elems; i++) + user_param->scale_img_height[i] = ctrl->p_new.p_u16[i]; + ctrl =v4l2_ctrl_find (hdl,V4L2_CID_MTK_FD_SCALE_IMG_NUM ); +if (ctrl) + user_param->scale_img_num = ctrl->val; + + ctrl =v4l2_ctrl_find (hdl,V4L2_CID_MTK_FD_DETECT_POSE ); +if (ctrl) +for (i =0 ; i < ctrl->elems; i++) + user_param->face_directions[i] = ctrl->p_new.p_u16[i]; + ctrl =v4l2_ctrl_find (hdl,V4L2_CID_MTK_FD_DETECT_SPEED ); +if (ctrl) + user_param->fd_speedup = ctrl->val; + ctrl =v4l2_ctrl_find (hdl,V4L2_CID_MTK_FD_DETECTION_MODEL ); +if (ctrl) + user_param->fd_extra_model = ctrl->val; +} + +static void mtk_fd_device_run (void *priv) +{ +struct mtk_fd_ctx *ctx = priv; +struct mtk_fd_dev *fd = ctx->fd_dev; +struct vb2_v4l2_buffer *src_buf, *dst_buf; +struct fd_enq_param fd_param; + + src_buf =v4l2_m2m_next_src_buf (ctx->fh.m2m_ctx); + dst_buf =v4l2_m2m_next_dst_buf (ctx->fh.m2m_ctx); + + fd_param.src_img[0 ].dma_addr = +vb2_dma_contig_plane_dma_addr (&src_buf->vb2_buf,0 ); + fd_param.user_result.dma_addr = +vb2_dma_contig_plane_dma_addr (&dst_buf->vb2_buf,0 ); + fd_param.output_vaddr = (u64 )vb2_plane_vaddr (&dst_buf->vb2_buf,0 ); + fd_param.user_param.src_img_fmt = +get_fd_img_fmt (ctx->src_fmt.pixelformat); +if (ctx->src_fmt.num_planes ==2 ) + fd_param.src_img[1 ].dma_addr = +vb2_dma_contig_plane_dma_addr (&src_buf->vb2_buf,1 ); +mtk_fd_fill_user_param (&fd_param.user_param, &ctx->hdl); + +/* Complete request controls if any */ +v4l2_ctrl_request_complete (src_buf->vb2_buf.req_obj.req, &ctx->hdl); + + fd->output = (struct fd_user_output *)fd_param.output_vaddr; +mtk_fd_hw_job_exec (fd, &fd_param); +} + +static struct v4l2_m2m_ops fd_m2m_ops = { + .device_run = mtk_fd_device_run, +}; + +static const struct media_device_ops fd_m2m_media_ops = { + .req_validate = vb2_request_validate, + .req_queue = v4l2_m2m_request_queue, +};
[snip]
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c line 1142
Quoted 12 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +12
-0+ +static int mtk_fd_suspend (struct device *dev) +{ +struct mtk_fd_dev *fd =dev_get_drvdata (dev); + +if (pm_runtime_suspended (dev)) +return 0 ; + +if (fd->fd_stream_count) +if (mtk_fd_job_abort (fd)) +mtk_fd_hw_job_finish (fd,VB2_BUF_STATE_ERROR ); +
To avoid mtk_fd_hw_job_finish() trigger the next job,
I suppose that we could use v4l2_m2m_cancel_job instead of job_abort and job_finish here.
/**
* v4l2_m2m_cancel_job() - cancel pending jobs for the context
* @m2m_ctx: m2m context with jobs to be canceled
*
* In case of streamoff or release called on any context,
* 1] If the context is currently running, then abort job will be called
* 2] If the context is queued, then the context will be removed from
* the job_queue
*/
or another way,
we may add a flag and implement mtk_fd_job_ready() that reads the flag if we suspend, we set the flag and do job_abort and job_finish, even if it try enqueue, it will still not really queue the job, until we reset the flag in mtk_fd_resume().
how do you think?
Quoted 141 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +77
-0+/* suspend FD HW */ +writel (0x0 , fd->fd_base +MTK_FD_REG_OFFSET_INT_EN ); +writel (0x0 , fd->fd_base +MTK_FD_REG_OFFSET_HW_ENABLE ); +clk_disable_unprepare (fd->fd_clk); +dev_dbg (dev,"%s:disable clock\n" , __func__); + +return 0 ; +} + +static int mtk_fd_resume (struct device *dev) +{ +struct mtk_fd_dev *fd =dev_get_drvdata (dev); +int ret; + +if (pm_runtime_suspended (dev)) +return 0 ; + + ret =clk_prepare_enable (fd->fd_clk); +if (ret <0 ) { +dev_dbg (dev,"Failed to open fd clk:%d\n" , ret); +return ret; + } + +/* resume FD HW */ +writel (MTK_FD_SET_HW_ENABLE , fd->fd_base +MTK_FD_REG_OFFSET_HW_ENABLE ); +writel (0x1 , fd->fd_base +MTK_FD_REG_OFFSET_INT_EN ); +dev_dbg (dev,"%s:enable clock\n" , __func__); + +return 0 ; +} + +static int mtk_fd_runtime_suspend (struct device *dev) +{ +struct mtk_fd_dev *fd =dev_get_drvdata (dev); + +clk_disable_unprepare (fd->fd_clk); +return 0 ; +} + +static int mtk_fd_runtime_resume (struct device *dev) +{ +struct mtk_fd_dev *fd =dev_get_drvdata (dev); +int ret; + + ret =clk_prepare_enable (fd->fd_clk); +if (ret <0 ) { +dev_err (dev,"Failed to open fd clk:%d\n" , ret); +return ret; + } + +return 0 ; +} + +static const struct dev_pm_ops mtk_fd_pm_ops = { + SET_SYSTEM_SLEEP_PM_OPS(mtk_fd_suspend, mtk_fd_resume) +SET_RUNTIME_PM_OPS (mtk_fd_runtime_suspend, mtk_fd_runtime_resume,NULL ) +}; + +static const struct of_device_id mtk_fd_of_ids[] = { + { .compatible ="mediatek,mt8183-fd" , }, + {} +}; +MODULE_DEVICE_TABLE (of, mtk_fd_of_ids); + +static struct platform_driver mtk_fd_driver = { + .probe = mtk_fd_probe, + .remove = mtk_fd_remove, + .driver = { + .name ="mtk-fd-4.0" , + .of_match_table =of_match_ptr (mtk_fd_of_ids), + .pm = &mtk_fd_pm_ops, + } +}; +module_platform_driver (mtk_fd_driver); + +MODULE_DESCRIPTION ("Mediatek FD driver" ); +MODULE_LICENSE ("GPL" );include/uapi/linux/mtk-fd-v4l2-controls.h new file +58
-0@@ -0,0 +1,69 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +// +// Copyright (c) 2019 MediaTek Inc. +/* +* For V4L2_CID_MTK_FD_DETECT_POSE, User can set the desired face direction to +* be detected for each face angle, there are five face angle and 12 directions. +* Below shows the definition of face angle and face direction, +* and a recommended usage of for face detection, the more selected directions +* the longer HW process time needed. +* +* enum face_angle { +* MTK_FD_FACE_FRONT, +* MTK_FD_FACE_RIGHT_50, +* MTK_FD_FACE_LEFT_50, +* MTK_FD_FACE_RIGHT_90, +* MTK_FD_FACE_LEFT_90, +* MTK_FD_FACE_ANGLE_NUM, +* }; +* +* struct face_direction_def { +* __u16 MTK_FD_FACE_DIR_0 : 1, +* MTK_FD_FACE_DIR_30 : 1, +* MTK_FD_FACE_DIR_60 : 1, +* MTK_FD_FACE_DIR_90 : 1, +* MTK_FD_FACE_DIR_120 : 1, +* MTK_FD_FACE_DIR_150 : 1, +* MTK_FD_FACE_DIR_180 : 1, +* MTK_FD_FACE_DIR_210 : 1, +* MTK_FD_FACE_DIR_240 : 1, +* MTK_FD_FACE_DIR_270 : 1, +* MTK_FD_FACE_DIR_300 : 1, +* MTK_FD_FACE_DIR_330 : 1, +* : 4; +* }; +* +* Sample usage: +* u16 face_directions[MTK_FD_FACE_ANGLE_NUM] = {0}; +* +* face_directions[MTK_FD_FACE_FRONT] = 0x3ff; +* +*/ + +#ifndef __UAPI_MTK_FD_V4L2_CONTROLS_H__ +#define __UAPI_MTK_FD_V4L2_CONTROLS_H__ + +#include <linux/v4l2-controls.h> + +/* Set the face angle and directions to be detected */ +#define V4L2_CID_MTK_FD_DETECT_POSE (V4L2_CID_USER_MTK_FD_BASE + 1) + +/* Set image widths for an input image to be scaled down for face detection */ +#define V4L2_CID_MTK_FD_SCALE_DOWN_IMG_WIDTH (V4L2_CID_USER_MTK_FD_BASE + 2) + +/* Set image heights for an input image to be scaled down for face detection */ +#define V4L2_CID_MTK_FD_SCALE_DOWN_IMG_HEIGHT (V4L2_CID_USER_MTK_FD_BASE + 3) + +/* Set the length of scale down size array */ +#define V4L2_CID_MTK_FD_SCALE_IMG_NUM (V4L2_CID_USER_MTK_FD_BASE + 4)
include/uapi/linux/mtk-fd-v4l2-controls.h line 68
Quoted 10 lines
include/uapi/linux/mtk-fd-v4l2-controls.h new file +10
-0+ +/* Set the detection speed, usually reducing accuracy. */ +#define V4L2_CID_MTK_FD_DETECT_SPEED (V4L2_CID_USER_MTK_FD_BASE + 5) + +/* Select the detection model or algorithm to be used. */ +#define V4L2_CID_MTK_FD_DETECTION_MODEL (V4L2_CID_USER_MTK_FD_BASE + 6) + +/* We reserve 16 controls for this driver. */ +#define V4L2_CID_MTK_FD_MAX 16 +
For these control IDs, I think the following should be remained as chip specific controls.V4L2_CID_MTK_FD_SCALE_DOWN_IMG_WIDTH, V4L2_CID_MTK_FD_SCALE_DOWN_IMG_HEIGHT and V4L2_CID_MTK_FD_SCALE_IMG_NUM
Hope there would be standardizing face detection api that cover the rest
controls: V4L2_CID_MTK_FD_DETECT_POSE, V4L2_CID_MTK_FD_DETECT_SPEED and V4L2_CID_MTK_FD_DETECTION_MODEL
Would you have any suggestions on how to propose the standard face detection apis?
Thanks and best regards
Jerry
Quoted 16 lines
include/uapi/linux/mtk-fd-v4l2-controls.h new file +1
-0+#endif /* __UAPI_MTK_FD_V4L2_CONTROLS_H__ */ include/uapi/linux/v4l2-controls.h +4
-0@@ -192,6 +192,10 @@ enum v4l2_colorfx {* We reserve 16 controls for this driver. */ #define V4L2_CID_USER_IMX_BASE (V4L2_CID_USER_BASE + 0x10b0) +/* The base for the mediatek FD driver controls */ +/* We reserve 16 controls for this driver. */ +#define V4L2_CID_USER_MTK_FD_BASE (V4L2_CID_USER_BASE + 0x10d0) +/* MPEG-class control IDs */ /* The MPEG controls are applicable to all codec controls * and the 'MPEG' part of the define is historical */
Re: [RFC PATCH V3 3/3] platform: mtk-isp: Add Mediatek FD driver
Quoted 30 lines · Jerry-ch Chen
Hi Tomasz,
Quoted 27 lines · Jerry-ch Chen
From: Jerry-ch Chen <[email protected]>
This patch adds the driver of Face Detection (FD) unit in Mediatek camera system, providing face detection function.
The mtk-isp directory will contain drivers for multiple IP blocks found in Mediatek ISP system. It will include ISP Pass 1 driver (CAM), sensor interface driver, DIP driver and face detection driver.
Signed-off-by Jerry-ch Chen <[email protected]>--- drivers/media/platform/Kconfig | 2 + drivers/media/platform/Makefile | 2 + drivers/media/platform/mtk-isp/fd/Kconfig | 19 + drivers/media/platform/mtk-isp/fd/Makefile | 5 + drivers/media/platform/mtk-isp/fd/mtk_fd.h | 148 ++ drivers/media/platform/mtk-isp/fd/mtk_fd_40.c | 1219 +++++++++++++++++ include/uapi/linux/mtk-fd-v4l2-controls.h | 69 + include/uapi/linux/v4l2-controls.h | 4 + 8 files changed, 1468 insertions(+) create mode 100644 drivers/media/platform/mtk-isp/fd/Kconfig create mode 100644 drivers/media/platform/mtk-isp/fd/Makefile create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd.h create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd_40.c create mode 100644 include/uapi/linux/mtk-fd-v4l2-controls.h
[snip]
Quoted 39 lines
Quoted 9 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +9
-0+static int mtk_fd_job_abort (struct mtk_fd_dev *fd) +{ +u32 ret; + + ret =wait_for_completion_timeout (&fd->fd_irq_done, +msecs_to_jiffies (MTK_FD_HW_TIMEOUT )); +/* Reset FD HW */ +if (!ret) { +struct ipi_message fd_ipi_msg;drivers/media/platform/mtk-isp/fd/mtk_fd_40.c line 362
Quoted 10 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +10
-0+ + fd_ipi_msg.cmd_id =MTK_FD_IPI_CMD_RESET ; +if (scp_ipi_send (fd->scp_pdev,SCP_IPI_FD_CMD , &fd_ipi_msg, +sizeof (fd_ipi_msg),MTK_FD_IPI_SEND_TIMEOUT )) +dev_err (fd->dev,"FD Reset HW error\n" ); +return -ETIMEDOUT ; + } +return 0 ; +} +Continue the discussion about job abort in RFC v2,
I think the
job_abortcallback inv4l2_m2m_ops()might be useful.ref:
https://elixir.bootlin.com/linux/v5.4-rc2/source/drivers/media/v4l2-core/v4l2-mem2mem.c#L398
https://elixir.bootlin.com/linux/v5.4-rc2/source/include/media/v4l2-mem2mem.h#L43in
drivers/media/v4l2-core/v4l2-mem2mem.c#398v4l2_m2m_cancel_job()
...if (m2m_ctx->job_flags & TRANS_RUNNING) { spin_unlock_irqrestore(&m2m_dev->job_spinlock, flags); if (m2m_dev->m2m_ops->job_abort) m2m_dev->m2m_ops->job_abort(m2m_ctx->priv); dprintk("m2m_ctx %p running, will wait to complete\n", m2m_ctx); wait_event(m2m_ctx->finished, !(m2m_ctx->job_flags & TRANS_RUNNING)); } ...
Quoted 3 lines
If this operation is set, we might use the
v4l2_m2m_cancel_job()when suspend, and it will domtk_fd_job_abort()here.
The expectation for .job_abort() is that signals the hardware to instantly abandon the current job. Do we have a way to tell the firmware/hardware to do so?
Also, suspend must not abort the current job. Anything that was already running is expected to complete successfuly and further jobs should continue executing once the system resumes.
[snip]
Quoted 27 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c line 1142
Quoted 12 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +12
-0+ +static int mtk_fd_suspend (struct device *dev) +{ +struct mtk_fd_dev *fd =dev_get_drvdata (dev); + +if (pm_runtime_suspended (dev)) +return 0 ; + +if (fd->fd_stream_count) +if (mtk_fd_job_abort (fd)) +mtk_fd_hw_job_finish (fd,VB2_BUF_STATE_ERROR ); +To avoid
mtk_fd_hw_job_finish()trigger the next job,
I suppose that we could usev4l2_m2m_cancel_jobinstead ofjob_abortandjob_finishhere./**
*v4l2_m2m_cancel_job()- cancel pending jobs for the context
* @m2m_ctx: m2m context with jobs to be canceled
*
* In case of streamoff or release called on any context,
* 1] If the context is currently running, then abort job will be called
* 2] If the context is queued, then the context will be removed from
* thejob_queue
*/
Quoted 8 lines
or another way,
we may add a flag and implementmtk_fd_job_ready()that reads the flag if we suspend, we set the flag and dojob_abortandjob_finish, even if it try enqueue, it will still not really queue the job, until we reset the flag inmtk_fd_resume().how do you think?
As per my comment above, suspend must just pause the execution of the jobs. It must not cause any jobs to be skipped.
After analyzing the m2m framework and existing m2m drivers I realized that they currently provide no way to correctly handle suspend/resume. Pi-Hsun has been looking into fixing this in crrev.com/c/1878112 and we'll send it upstream as soon as we get something that should handle all the cases correctly.
Quoted 19 lines
Quoted 19 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +19
-0+/* suspend FD HW */ +writel (0x0 , fd->fd_base +MTK_FD_REG_OFFSET_INT_EN ); +writel (0x0 , fd->fd_base +MTK_FD_REG_OFFSET_HW_ENABLE ); +clk_disable_unprepare (fd->fd_clk); +dev_dbg (dev,"%s:disable clock\n" , __func__); + +return 0 ; +} + +static int mtk_fd_resume (struct device *dev) +{ +struct mtk_fd_dev *fd =dev_get_drvdata (dev); +int ret; + +if (pm_runtime_suspended (dev)) +return 0 ; + + ret =clk_prepare_enable (fd->fd_clk); +if (ret <0 ) {
Quoted 8 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c line 1169
Quoted 8 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +8
-0+dev_dbg (dev,"Failed to open fd clk:%d\n" , ret); +return ret; + } + +/* resume FD HW */ +writel (MTK_FD_SET_HW_ENABLE , fd->fd_base +MTK_FD_REG_OFFSET_HW_ENABLE ); +writel (0x1 , fd->fd_base +MTK_FD_REG_OFFSET_INT_EN ); +dev_dbg (dev,"%s:enable clock\n" , __func__);
By the way, we need to kick the m2m framework here to schedule further jobs. Pi-Hsun's patch will also take care of this.
[snip]
Quoted 27 lines
Quoted 11 lines
include/uapi/linux/mtk-fd-v4l2-controls.h new file +11
-0+/* Set the face angle and directions to be detected */ +#define V4L2_CID_MTK_FD_DETECT_POSE (V4L2_CID_USER_MTK_FD_BASE + 1) + +/* Set image widths for an input image to be scaled down for face detection */ +#define V4L2_CID_MTK_FD_SCALE_DOWN_IMG_WIDTH (V4L2_CID_USER_MTK_FD_BASE + 2) + +/* Set image heights for an input image to be scaled down for face detection */ +#define V4L2_CID_MTK_FD_SCALE_DOWN_IMG_HEIGHT (V4L2_CID_USER_MTK_FD_BASE + 3) + +/* Set the length of scale down size array */ +#define V4L2_CID_MTK_FD_SCALE_IMG_NUM (V4L2_CID_USER_MTK_FD_BASE + 4)include/uapi/linux/mtk-fd-v4l2-controls.h line 68
Quoted 10 lines
include/uapi/linux/mtk-fd-v4l2-controls.h new file +10
-0+ +/* Set the detection speed, usually reducing accuracy. */ +#define V4L2_CID_MTK_FD_DETECT_SPEED (V4L2_CID_USER_MTK_FD_BASE + 5) + +/* Select the detection model or algorithm to be used. */ +#define V4L2_CID_MTK_FD_DETECTION_MODEL (V4L2_CID_USER_MTK_FD_BASE + 6) + +/* We reserve 16 controls for this driver. */ +#define V4L2_CID_MTK_FD_MAX 16 +For these control IDs, I think the following should be remained as chip specific controls.
V4L2_CID_MTK_FD_SCALE_DOWN_IMG_WIDTH,V4L2_CID_MTK_FD_SCALE_DOWN_IMG_HEIGHTandV4L2_CID_MTK_FD_SCALE_IMG_NUM
Quoted 7 lines
Hope there would be standardizing face detection api that cover the rest
controls:V4L2_CID_MTK_FD_DETECT_POSE,V4L2_CID_MTK_FD_DETECT_SPEEDandV4L2_CID_MTK_FD_DETECTION_MODELWould you have any suggestions on how to propose the standard face detection apis?
Given no follow up feedback from the community, I think we can keep them as driver-specific, but should make sure that they have some reasonable default values in case an application doesn't recognize them.
Best regards,
Tomasz
Re: [RFC PATCH V3 3/3] platform: mtk-isp: Add Mediatek FD driver
Hi Tomasz,
Quoted 77 lines · Tomasz Figa
Quoted 30 lines · Jerry-ch Chen
Hi Tomasz,
Quoted 27 lines · Jerry-ch Chen
From: Jerry-ch Chen <[email protected]>
This patch adds the driver of Face Detection (FD) unit in Mediatek camera system, providing face detection function.
The mtk-isp directory will contain drivers for multiple IP blocks found in Mediatek ISP system. It will include ISP Pass 1 driver (CAM), sensor interface driver, DIP driver and face detection driver.
Signed-off-by Jerry-ch Chen <[email protected]>--- drivers/media/platform/Kconfig | 2 + drivers/media/platform/Makefile | 2 + drivers/media/platform/mtk-isp/fd/Kconfig | 19 + drivers/media/platform/mtk-isp/fd/Makefile | 5 + drivers/media/platform/mtk-isp/fd/mtk_fd.h | 148 ++ drivers/media/platform/mtk-isp/fd/mtk_fd_40.c | 1219 +++++++++++++++++ include/uapi/linux/mtk-fd-v4l2-controls.h | 69 + include/uapi/linux/v4l2-controls.h | 4 + 8 files changed, 1468 insertions(+) create mode 100644 drivers/media/platform/mtk-isp/fd/Kconfig create mode 100644 drivers/media/platform/mtk-isp/fd/Makefile create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd.h create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd_40.c create mode 100644 include/uapi/linux/mtk-fd-v4l2-controls.h[snip]
Quoted 39 lines
Quoted 9 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +9
-0+static int mtk_fd_job_abort (struct mtk_fd_dev *fd) +{ +u32 ret; + + ret =wait_for_completion_timeout (&fd->fd_irq_done, +msecs_to_jiffies (MTK_FD_HW_TIMEOUT )); +/* Reset FD HW */ +if (!ret) { +struct ipi_message fd_ipi_msg;drivers/media/platform/mtk-isp/fd/mtk_fd_40.c line 362
Quoted 10 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +10
-0+ + fd_ipi_msg.cmd_id =MTK_FD_IPI_CMD_RESET ; +if (scp_ipi_send (fd->scp_pdev,SCP_IPI_FD_CMD , &fd_ipi_msg, +sizeof (fd_ipi_msg),MTK_FD_IPI_SEND_TIMEOUT )) +dev_err (fd->dev,"FD Reset HW error\n" ); +return -ETIMEDOUT ; + } +return 0 ; +} +Continue the discussion about job abort in RFC v2,
I think the
job_abortcallback inv4l2_m2m_ops()might be useful.ref:
https://elixir.bootlin.com/linux/v5.4-rc2/source/drivers/media/v4l2-core/v4l2-mem2mem.c#L398
https://elixir.bootlin.com/linux/v5.4-rc2/source/include/media/v4l2-mem2mem.h#L43in
drivers/media/v4l2-core/v4l2-mem2mem.c#398v4l2_m2m_cancel_job()
...if (m2m_ctx->job_flags & TRANS_RUNNING) { spin_unlock_irqrestore(&m2m_dev->job_spinlock, flags); if (m2m_dev->m2m_ops->job_abort) m2m_dev->m2m_ops->job_abort(m2m_ctx->priv); dprintk("m2m_ctx %p running, will wait to complete\n", m2m_ctx); wait_event(m2m_ctx->finished, !(m2m_ctx->job_flags & TRANS_RUNNING)); } ...Quoted 3 lines
If this operation is set, we might use the
v4l2_m2m_cancel_job()when suspend, and it will domtk_fd_job_abort()here.
Quoted 8 lines
The expectation for .job_abort() is that signals the hardware to instantly abandon the current job. Do we have a way to tell the
firmware/hardwareto do so?Also, suspend must not abort the current job. Anything that was already running is expected to complete successfuly and further jobs should continue executing once the system resumes.
I appreciate your comments and Pi-Hsun's patch,
Ok, I see.
For FD40, we can't tell the firmware/hardware to instantly abandon the current job.
Therefore, for suspend, we stop sending further jobs to hardware and wait for the completion of the running job.
For resume, we continue sending jobs to hardware.
Quoted 38 lines
[snip]
Quoted 27 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c line 1142
Quoted 12 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +12
-0+ +static int mtk_fd_suspend (struct device *dev) +{ +struct mtk_fd_dev *fd =dev_get_drvdata (dev); + +if (pm_runtime_suspended (dev)) +return 0 ; + +if (fd->fd_stream_count) +if (mtk_fd_job_abort (fd)) +mtk_fd_hw_job_finish (fd,VB2_BUF_STATE_ERROR ); +To avoid
mtk_fd_hw_job_finish()trigger the next job,
I suppose that we could usev4l2_m2m_cancel_jobinstead ofjob_abortandjob_finishhere./**
*v4l2_m2m_cancel_job()- cancel pending jobs for the context
* @m2m_ctx: m2m context with jobs to be canceled
*
* In case of streamoff or release called on any context,
* 1] If the context is currently running, then abort job will be called
* 2] If the context is queued, then the context will be removed from
* thejob_queue
*/Quoted 8 lines
or another way,
we may add a flag and implementmtk_fd_job_ready()that reads the flag if we suspend, we set the flag and dojob_abortandjob_finish, even if it try enqueue, it will still not really queue the job, until we reset the flag inmtk_fd_resume().how do you think?
Quoted 9 lines
As per my comment above, suspend must just pause the execution of the jobs. It must not cause any jobs to be skipped.
After analyzing the m2m framework and existing m2m drivers I realized that they currently provide no way to correctly handle suspend/resume. Pi-Hsun has been looking into fixing this in crrev.com/c/1878112 and we'll send it upstream as soon as we get something that should handle all the cases correctly.
Ok, thanks for the patches.
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +22
-0+/* suspend FD HW */ +writel (0x0 , fd->fd_base +MTK_FD_REG_OFFSET_INT_EN ); +writel (0x0 , fd->fd_base +MTK_FD_REG_OFFSET_HW_ENABLE ); +clk_disable_unprepare (fd->fd_clk); +dev_dbg (dev,"%s:disable clock\n" , __func__); + +return 0 ; +} + +static int mtk_fd_resume (struct device *dev) +{ +struct mtk_fd_dev *fd =dev_get_drvdata (dev); +int ret; + +if (pm_runtime_suspended (dev)) +return 0 ; + + ret =clk_prepare_enable (fd->fd_clk); +if (ret <0 ) { +dev_dbg (dev,"Failed to open fd clk:%d\n" , ret); +return ret; + }
Quoted 9 lines
Quoted 5 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c line 1169
Quoted 5 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +5
-0+ +/* resume FD HW */ +writel (MTK_FD_SET_HW_ENABLE , fd->fd_base +MTK_FD_REG_OFFSET_HW_ENABLE ); +writel (0x1 , fd->fd_base +MTK_FD_REG_OFFSET_INT_EN ); +dev_dbg (dev,"%s:enable clock\n" , __func__);By the way, we need to kick the m2m framework here to schedule further jobs. Pi-Hsun's patch will also take care of this.
Ok, I see.
I would like to use Pi-Hsun's patch, otherwise I would need to call v4l2_m2m_try_run() here.
Quoted 37 lines
[snip]
Quoted 27 lines
Quoted 11 lines
include/uapi/linux/mtk-fd-v4l2-controls.h new file +11
-0+/* Set the face angle and directions to be detected */ +#define V4L2_CID_MTK_FD_DETECT_POSE (V4L2_CID_USER_MTK_FD_BASE + 1) + +/* Set image widths for an input image to be scaled down for face detection */ +#define V4L2_CID_MTK_FD_SCALE_DOWN_IMG_WIDTH (V4L2_CID_USER_MTK_FD_BASE + 2) + +/* Set image heights for an input image to be scaled down for face detection */ +#define V4L2_CID_MTK_FD_SCALE_DOWN_IMG_HEIGHT (V4L2_CID_USER_MTK_FD_BASE + 3) + +/* Set the length of scale down size array */ +#define V4L2_CID_MTK_FD_SCALE_IMG_NUM (V4L2_CID_USER_MTK_FD_BASE + 4)include/uapi/linux/mtk-fd-v4l2-controls.h line 68
Quoted 10 lines
include/uapi/linux/mtk-fd-v4l2-controls.h new file +10
-0+ +/* Set the detection speed, usually reducing accuracy. */ +#define V4L2_CID_MTK_FD_DETECT_SPEED (V4L2_CID_USER_MTK_FD_BASE + 5) + +/* Select the detection model or algorithm to be used. */ +#define V4L2_CID_MTK_FD_DETECTION_MODEL (V4L2_CID_USER_MTK_FD_BASE + 6) + +/* We reserve 16 controls for this driver. */ +#define V4L2_CID_MTK_FD_MAX 16 +For these control IDs, I think the following should be remained as chip specific controls.
V4L2_CID_MTK_FD_SCALE_DOWN_IMG_WIDTH,V4L2_CID_MTK_FD_SCALE_DOWN_IMG_HEIGHTandV4L2_CID_MTK_FD_SCALE_IMG_NUMQuoted 7 lines
Hope there would be standardizing face detection api that cover the rest
controls:V4L2_CID_MTK_FD_DETECT_POSE,V4L2_CID_MTK_FD_DETECT_SPEEDandV4L2_CID_MTK_FD_DETECTION_MODELWould you have any suggestions on how to propose the standard face detection apis?
Quoted 7 lines
Given no follow up feedback from the community, I think we can keep them as driver-specific, but should make sure that they have some reasonable default values in case an application doesn't recognize them.
Best regards,
Tomasz
Should I keep the file "mtk-fd-v4l2-controls.h" which defines the control ids under the folder "/include/uapi/linux"?
Thanks and Best regards,
Jerry
Re: [RFC PATCH V3 3/3] platform: mtk-isp: Add Mediatek FD driver
Quoted 178 lines · Jerry-ch Chen
Hi Tomasz,
Quoted 77 lines · Tomasz Figa
Quoted 30 lines · Jerry-ch Chen
Hi Tomasz,
Quoted 27 lines · Jerry-ch Chen
From: Jerry-ch Chen <[email protected]>
This patch adds the driver of Face Detection (FD) unit in Mediatek camera system, providing face detection function.
The mtk-isp directory will contain drivers for multiple IP blocks found in Mediatek ISP system. It will include ISP Pass 1 driver (CAM), sensor interface driver, DIP driver and face detection driver.
Signed-off-by Jerry-ch Chen <[email protected]>--- drivers/media/platform/Kconfig | 2 + drivers/media/platform/Makefile | 2 + drivers/media/platform/mtk-isp/fd/Kconfig | 19 + drivers/media/platform/mtk-isp/fd/Makefile | 5 + drivers/media/platform/mtk-isp/fd/mtk_fd.h | 148 ++ drivers/media/platform/mtk-isp/fd/mtk_fd_40.c | 1219 +++++++++++++++++ include/uapi/linux/mtk-fd-v4l2-controls.h | 69 + include/uapi/linux/v4l2-controls.h | 4 + 8 files changed, 1468 insertions(+) create mode 100644 drivers/media/platform/mtk-isp/fd/Kconfig create mode 100644 drivers/media/platform/mtk-isp/fd/Makefile create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd.h create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd_40.c create mode 100644 include/uapi/linux/mtk-fd-v4l2-controls.h[snip]
Quoted 39 lines
Quoted 9 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +9
-0+static int mtk_fd_job_abort (struct mtk_fd_dev *fd) +{ +u32 ret; + + ret =wait_for_completion_timeout (&fd->fd_irq_done, +msecs_to_jiffies (MTK_FD_HW_TIMEOUT )); +/* Reset FD HW */ +if (!ret) { +struct ipi_message fd_ipi_msg;drivers/media/platform/mtk-isp/fd/mtk_fd_40.c line 362
Quoted 10 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +10
-0+ + fd_ipi_msg.cmd_id =MTK_FD_IPI_CMD_RESET ; +if (scp_ipi_send (fd->scp_pdev,SCP_IPI_FD_CMD , &fd_ipi_msg, +sizeof (fd_ipi_msg),MTK_FD_IPI_SEND_TIMEOUT )) +dev_err (fd->dev,"FD Reset HW error\n" ); +return -ETIMEDOUT ; + } +return 0 ; +} +Continue the discussion about job abort in RFC v2,
I think the
job_abortcallback inv4l2_m2m_ops()might be useful.ref:
https://elixir.bootlin.com/linux/v5.4-rc2/source/drivers/media/v4l2-core/v4l2-mem2mem.c#L398
https://elixir.bootlin.com/linux/v5.4-rc2/source/include/media/v4l2-mem2mem.h#L43in
drivers/media/v4l2-core/v4l2-mem2mem.c#398v4l2_m2m_cancel_job()
...if (m2m_ctx->job_flags & TRANS_RUNNING) { spin_unlock_irqrestore(&m2m_dev->job_spinlock, flags); if (m2m_dev->m2m_ops->job_abort) m2m_dev->m2m_ops->job_abort(m2m_ctx->priv); dprintk("m2m_ctx %p running, will wait to complete\n", m2m_ctx); wait_event(m2m_ctx->finished, !(m2m_ctx->job_flags & TRANS_RUNNING)); } ...Quoted 3 lines
If this operation is set, we might use the
v4l2_m2m_cancel_job()when suspend, and it will domtk_fd_job_abort()here.Quoted 8 lines
The expectation for .job_abort() is that signals the hardware to instantly abandon the current job. Do we have a way to tell the
firmware/hardwareto do so?Also, suspend must not abort the current job. Anything that was already running is expected to complete successfuly and further jobs should continue executing once the system resumes.
I appreciate your comments and Pi-Hsun's patch,
Ok, I see.
For FD40, we can't tell thefirmware/hardwareto instantly abandon the current job.
Therefore, for suspend, we stop sending further jobs to hardware and wait for the completion of the running job.
For resume, we continue sending jobs to hardware.Quoted 38 lines
[snip]
Quoted 27 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c line 1142
Quoted 12 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +12
-0+ +static int mtk_fd_suspend (struct device *dev) +{ +struct mtk_fd_dev *fd =dev_get_drvdata (dev); + +if (pm_runtime_suspended (dev)) +return 0 ; + +if (fd->fd_stream_count) +if (mtk_fd_job_abort (fd)) +mtk_fd_hw_job_finish (fd,VB2_BUF_STATE_ERROR ); +To avoid
mtk_fd_hw_job_finish()trigger the next job,
I suppose that we could usev4l2_m2m_cancel_jobinstead ofjob_abortandjob_finishhere./**
*v4l2_m2m_cancel_job()- cancel pending jobs for the context
* @m2m_ctx: m2m context with jobs to be canceled
*
* In case of streamoff or release called on any context,
* 1] If the context is currently running, then abort job will be called
* 2] If the context is queued, then the context will be removed from
* thejob_queue
*/Quoted 8 lines
or another way,
we may add a flag and implementmtk_fd_job_ready()that reads the flag if we suspend, we set the flag and dojob_abortandjob_finish, even if it try enqueue, it will still not really queue the job, until we reset the flag inmtk_fd_resume().how do you think?
Quoted 9 lines
As per my comment above, suspend must just pause the execution of the jobs. It must not cause any jobs to be skipped.
After analyzing the m2m framework and existing m2m drivers I realized that they currently provide no way to correctly handle suspend/resume. Pi-Hsun has been looking into fixing this in crrev.com/c/1878112 and we'll send it upstream as soon as we get something that should handle all the cases correctly.
Ok, thanks for the patches.
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
+ /* suspend FD HW */
+writel(0x0, fd->fd_base + MTK_FD_REG_OFFSET_INT_EN);
+writel(0x0, fd->fd_base + MTK_FD_REG_OFFSET_HW_ENABLE);
+clk_disable_unprepare(fd->fd_clk);
+dev_dbg(dev, "%s:disable clock\n", __func__);drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +17
-0+ +return 0 ; +} + +static int mtk_fd_resume (struct device *dev) +{ +struct mtk_fd_dev *fd =dev_get_drvdata (dev); +int ret; + +if (pm_runtime_suspended (dev)) +return 0 ; + + ret =clk_prepare_enable (fd->fd_clk); +if (ret <0 ) { +dev_dbg (dev,"Failed to open fd clk:%d\n" , ret); +return ret; + }Quoted 9 lines
Quoted 5 lines
Quoted 5 lines
+
+ /* resume FD HW */
+writel(MTK_FD_SET_HW_ENABLE, fd->fd_base + MTK_FD_REG_OFFSET_HW_ENABLE);
+writel(0x1, fd->fd_base + MTK_FD_REG_OFFSET_INT_EN);
+dev_dbg(dev, "%s:enable clock\n", __func__);By the way, we need to kick the m2m framework here to schedule further jobs. Pi-Hsun's patch will also take care of this.
Quoted 4 lines
Ok, I see.
I would like to use Pi-Hsun's patch, otherwise I would need to callv4l2_m2m_try_run()here.
Yes, please include Pi-Hsun's patch (with original author, sign-off + your sign-off added) at the beginning of the next version of your series.
Quoted 41 lines
Quoted 41 lines
[snip]
Quoted 27 lines
Quoted 11 lines
include/uapi/linux/mtk-fd-v4l2-controls.h new file +11
-0+/* Set the face angle and directions to be detected */ +#define V4L2_CID_MTK_FD_DETECT_POSE (V4L2_CID_USER_MTK_FD_BASE + 1) + +/* Set image widths for an input image to be scaled down for face detection */ +#define V4L2_CID_MTK_FD_SCALE_DOWN_IMG_WIDTH (V4L2_CID_USER_MTK_FD_BASE + 2) + +/* Set image heights for an input image to be scaled down for face detection */ +#define V4L2_CID_MTK_FD_SCALE_DOWN_IMG_HEIGHT (V4L2_CID_USER_MTK_FD_BASE + 3) + +/* Set the length of scale down size array */ +#define V4L2_CID_MTK_FD_SCALE_IMG_NUM (V4L2_CID_USER_MTK_FD_BASE + 4)include/uapi/linux/mtk-fd-v4l2-controls.h line 68
Quoted 10 lines
include/uapi/linux/mtk-fd-v4l2-controls.h new file +10
-0+ +/* Set the detection speed, usually reducing accuracy. */ +#define V4L2_CID_MTK_FD_DETECT_SPEED (V4L2_CID_USER_MTK_FD_BASE + 5) + +/* Select the detection model or algorithm to be used. */ +#define V4L2_CID_MTK_FD_DETECTION_MODEL (V4L2_CID_USER_MTK_FD_BASE + 6) + +/* We reserve 16 controls for this driver. */ +#define V4L2_CID_MTK_FD_MAX 16 +For these control IDs, I think the following should be remained as chip specific controls.
V4L2_CID_MTK_FD_SCALE_DOWN_IMG_WIDTH,V4L2_CID_MTK_FD_SCALE_DOWN_IMG_HEIGHTandV4L2_CID_MTK_FD_SCALE_IMG_NUMQuoted 7 lines
Hope there would be standardizing face detection api that cover the rest
controls:V4L2_CID_MTK_FD_DETECT_POSE,V4L2_CID_MTK_FD_DETECT_SPEEDandV4L2_CID_MTK_FD_DETECTION_MODELWould you have any suggestions on how to propose the standard face detection apis?
Given no follow up feedback from the community, I think we can keep them as driver-specific, but should make sure that they have some reasonable default values in case an application doesn't recognize them.
Quoted 5 lines
Quoted 3 lines
Best regards,
TomaszShould I keep the file "
mtk-fd-v4l2-controls.h" which defines the control ids under the folder "/include/uapi/linux"?
We should define the CID base for the FD driver in v4l2-controls.h, but the controls themselves should be only defined inside the driver.
For example:
https://elixir.bootlin.com/linux/v5.4-rc8/source/include/uapi/linux/v4l2-controls.h#L178
https://elixir.bootlin.com/linux/v5.4-rc8/source/drivers/media/i2c/adv7180.c#L181
Best regards,
Tomasz
Re: [RFC PATCH V3 3/3] platform: mtk-isp: Add Mediatek FD driver
Hi Tomasz,
Quoted 180 lines · Tomasz Figa
Quoted 178 lines · Jerry-ch Chen
Hi Tomasz,
Quoted 77 lines · Tomasz Figa
Quoted 30 lines · Jerry-ch Chen
Hi Tomasz,
Quoted 27 lines · Jerry-ch Chen
From: Jerry-ch Chen <[email protected]>
This patch adds the driver of Face Detection (FD) unit in Mediatek camera system, providing face detection function.
The mtk-isp directory will contain drivers for multiple IP blocks found in Mediatek ISP system. It will include ISP Pass 1 driver (CAM), sensor interface driver, DIP driver and face detection driver.
Signed-off-by Jerry-ch Chen <[email protected]>--- drivers/media/platform/Kconfig | 2 + drivers/media/platform/Makefile | 2 + drivers/media/platform/mtk-isp/fd/Kconfig | 19 + drivers/media/platform/mtk-isp/fd/Makefile | 5 + drivers/media/platform/mtk-isp/fd/mtk_fd.h | 148 ++ drivers/media/platform/mtk-isp/fd/mtk_fd_40.c | 1219 +++++++++++++++++ include/uapi/linux/mtk-fd-v4l2-controls.h | 69 + include/uapi/linux/v4l2-controls.h | 4 + 8 files changed, 1468 insertions(+) create mode 100644 drivers/media/platform/mtk-isp/fd/Kconfig create mode 100644 drivers/media/platform/mtk-isp/fd/Makefile create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd.h create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd_40.c create mode 100644 include/uapi/linux/mtk-fd-v4l2-controls.h[snip]
Quoted 39 lines
Quoted 9 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +9
-0+static int mtk_fd_job_abort (struct mtk_fd_dev *fd) +{ +u32 ret; + + ret =wait_for_completion_timeout (&fd->fd_irq_done, +msecs_to_jiffies (MTK_FD_HW_TIMEOUT )); +/* Reset FD HW */ +if (!ret) { +struct ipi_message fd_ipi_msg;drivers/media/platform/mtk-isp/fd/mtk_fd_40.c line 362
Quoted 10 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +10
-0+ + fd_ipi_msg.cmd_id =MTK_FD_IPI_CMD_RESET ; +if (scp_ipi_send (fd->scp_pdev,SCP_IPI_FD_CMD , &fd_ipi_msg, +sizeof (fd_ipi_msg),MTK_FD_IPI_SEND_TIMEOUT )) +dev_err (fd->dev,"FD Reset HW error\n" ); +return -ETIMEDOUT ; + } +return 0 ; +} +Continue the discussion about job abort in RFC v2,
I think the
job_abortcallback inv4l2_m2m_ops()might be useful.ref:
https://elixir.bootlin.com/linux/v5.4-rc2/source/drivers/media/v4l2-core/v4l2-mem2mem.c#L398
https://elixir.bootlin.com/linux/v5.4-rc2/source/include/media/v4l2-mem2mem.h#L43in
drivers/media/v4l2-core/v4l2-mem2mem.c#398v4l2_m2m_cancel_job()
...if (m2m_ctx->job_flags & TRANS_RUNNING) { spin_unlock_irqrestore(&m2m_dev->job_spinlock, flags); if (m2m_dev->m2m_ops->job_abort) m2m_dev->m2m_ops->job_abort(m2m_ctx->priv); dprintk("m2m_ctx %p running, will wait to complete\n", m2m_ctx); wait_event(m2m_ctx->finished, !(m2m_ctx->job_flags & TRANS_RUNNING)); } ...Quoted 3 lines
If this operation is set, we might use the
v4l2_m2m_cancel_job()when suspend, and it will domtk_fd_job_abort()here.Quoted 8 lines
The expectation for .job_abort() is that signals the hardware to instantly abandon the current job. Do we have a way to tell the
firmware/hardwareto do so?Also, suspend must not abort the current job. Anything that was already running is expected to complete successfuly and further jobs should continue executing once the system resumes.
I appreciate your comments and Pi-Hsun's patch,
Ok, I see.
For FD40, we can't tell thefirmware/hardwareto instantly abandon the current job.
Therefore, for suspend, we stop sending further jobs to hardware and wait for the completion of the running job.
For resume, we continue sending jobs to hardware.Quoted 38 lines
[snip]
Quoted 27 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c line 1142
Quoted 12 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +12
-0+ +static int mtk_fd_suspend (struct device *dev) +{ +struct mtk_fd_dev *fd =dev_get_drvdata (dev); + +if (pm_runtime_suspended (dev)) +return 0 ; + +if (fd->fd_stream_count) +if (mtk_fd_job_abort (fd)) +mtk_fd_hw_job_finish (fd,VB2_BUF_STATE_ERROR ); +To avoid
mtk_fd_hw_job_finish()trigger the next job,
I suppose that we could usev4l2_m2m_cancel_jobinstead ofjob_abortandjob_finishhere./**
*v4l2_m2m_cancel_job()- cancel pending jobs for the context
* @m2m_ctx: m2m context with jobs to be canceled
*
* In case of streamoff or release called on any context,
* 1] If the context is currently running, then abort job will be called
* 2] If the context is queued, then the context will be removed from
* thejob_queue
*/Quoted 8 lines
or another way,
we may add a flag and implementmtk_fd_job_ready()that reads the flag if we suspend, we set the flag and dojob_abortandjob_finish, even if it try enqueue, it will still not really queue the job, until we reset the flag inmtk_fd_resume().how do you think?
Quoted 9 lines
As per my comment above, suspend must just pause the execution of the jobs. It must not cause any jobs to be skipped.
After analyzing the m2m framework and existing m2m drivers I realized that they currently provide no way to correctly handle suspend/resume. Pi-Hsun has been looking into fixing this in crrev.com/c/1878112 and we'll send it upstream as soon as we get something that should handle all the cases correctly.
Ok, thanks for the patches.
Quoted 31 lines
Quoted 19 lines
Quoted 19 lines
+ /* suspend FD HW */
+writel(0x0, fd->fd_base + MTK_FD_REG_OFFSET_INT_EN);
+writel(0x0, fd->fd_base + MTK_FD_REG_OFFSET_HW_ENABLE);
+clk_disable_unprepare(fd->fd_clk);
+dev_dbg(dev, "%s:disable clock\n", __func__);drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +14
-0+ +return 0 ; +} + +static int mtk_fd_resume (struct device *dev) +{ +struct mtk_fd_dev *fd =dev_get_drvdata (dev); +int ret; + +if (pm_runtime_suspended (dev)) +return 0 ; + + ret =clk_prepare_enable (fd->fd_clk); +if (ret <0 ) {Quoted 8 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c line 1169
Quoted 8 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +8
-0+dev_dbg (dev,"Failed to open fd clk:%d\n" , ret); +return ret; + } + +/* resume FD HW */ +writel (MTK_FD_SET_HW_ENABLE , fd->fd_base +MTK_FD_REG_OFFSET_HW_ENABLE ); +writel (0x1 , fd->fd_base +MTK_FD_REG_OFFSET_INT_EN ); +dev_dbg (dev,"%s:enable clock\n" , __func__);By the way, we need to kick the m2m framework here to schedule further jobs. Pi-Hsun's patch will also take care of this.
Quoted 9 lines
Quoted 4 lines
Ok, I see.
I would like to use Pi-Hsun's patch, otherwise I would need to callv4l2_m2m_try_run()here.Yes, please include Pi-Hsun's patch (with original author, sign-off + your sign-off added) at the beginning of the next version of your series.
Done.
Quoted 50 lines
Quoted 41 lines
Quoted 41 lines
[snip]
Quoted 27 lines
Quoted 11 lines
include/uapi/linux/mtk-fd-v4l2-controls.h new file +11
-0+/* Set the face angle and directions to be detected */ +#define V4L2_CID_MTK_FD_DETECT_POSE (V4L2_CID_USER_MTK_FD_BASE + 1) + +/* Set image widths for an input image to be scaled down for face detection */ +#define V4L2_CID_MTK_FD_SCALE_DOWN_IMG_WIDTH (V4L2_CID_USER_MTK_FD_BASE + 2) + +/* Set image heights for an input image to be scaled down for face detection */ +#define V4L2_CID_MTK_FD_SCALE_DOWN_IMG_HEIGHT (V4L2_CID_USER_MTK_FD_BASE + 3) + +/* Set the length of scale down size array */ +#define V4L2_CID_MTK_FD_SCALE_IMG_NUM (V4L2_CID_USER_MTK_FD_BASE + 4)include/uapi/linux/mtk-fd-v4l2-controls.h line 68
Quoted 10 lines
include/uapi/linux/mtk-fd-v4l2-controls.h new file +10
-0+ +/* Set the detection speed, usually reducing accuracy. */ +#define V4L2_CID_MTK_FD_DETECT_SPEED (V4L2_CID_USER_MTK_FD_BASE + 5) + +/* Select the detection model or algorithm to be used. */ +#define V4L2_CID_MTK_FD_DETECTION_MODEL (V4L2_CID_USER_MTK_FD_BASE + 6) + +/* We reserve 16 controls for this driver. */ +#define V4L2_CID_MTK_FD_MAX 16 +For these control IDs, I think the following should be remained as chip specific controls.
V4L2_CID_MTK_FD_SCALE_DOWN_IMG_WIDTH,V4L2_CID_MTK_FD_SCALE_DOWN_IMG_HEIGHTandV4L2_CID_MTK_FD_SCALE_IMG_NUMQuoted 7 lines
Hope there would be standardizing face detection api that cover the rest
controls:V4L2_CID_MTK_FD_DETECT_POSE,V4L2_CID_MTK_FD_DETECT_SPEEDandV4L2_CID_MTK_FD_DETECTION_MODELWould you have any suggestions on how to propose the standard face detection apis?
Given no follow up feedback from the community, I think we can keep them as driver-specific, but should make sure that they have some reasonable default values in case an application doesn't recognize them.
Quoted 5 lines
Quoted 3 lines
Best regards,
TomaszShould I keep the file "
mtk-fd-v4l2-controls.h" which defines the control ids under the folder "/include/uapi/linux"?We should define the CID base for the FD driver in
v4l2-controls.h, but the controls themselves should be only defined inside the driver.
Quoted 6 lines
For example:
https://elixir.bootlin.com/linux/v5.4-rc8/source/include/uapi/linux/v4l2-controls.h#L178
https://elixir.bootlin.com/linux/v5.4-rc8/source/drivers/media/i2c/adv7180.c#L181Best regards,
Tomasz
Appreciate for providing the example,
Would it be fine for me to put the private CID in the mtk_fd.h(which is similar to before...) or follow the example to define inside mtk_fd_40.c??
The next version is almost ready, maybe I can send it when I ready in a few days?
Thanks and best regards,
Jerry
Re: [RFC PATCH V3 3/3] platform: mtk-isp: Add Mediatek FD driver
Quoted 252 lines · Jerry-ch Chen
Hi Tomasz,
Quoted 180 lines · Tomasz Figa
Quoted 178 lines · Jerry-ch Chen
Hi Tomasz,
Quoted 77 lines · Tomasz Figa
Quoted 30 lines · Jerry-ch Chen
Hi Tomasz,
Quoted 27 lines · Jerry-ch Chen
From: Jerry-ch Chen <[email protected]>
This patch adds the driver of Face Detection (FD) unit in Mediatek camera system, providing face detection function.
The mtk-isp directory will contain drivers for multiple IP blocks found in Mediatek ISP system. It will include ISP Pass 1 driver (CAM), sensor interface driver, DIP driver and face detection driver.
Signed-off-by Jerry-ch Chen <[email protected]>--- drivers/media/platform/Kconfig | 2 + drivers/media/platform/Makefile | 2 + drivers/media/platform/mtk-isp/fd/Kconfig | 19 + drivers/media/platform/mtk-isp/fd/Makefile | 5 + drivers/media/platform/mtk-isp/fd/mtk_fd.h | 148 ++ drivers/media/platform/mtk-isp/fd/mtk_fd_40.c | 1219 +++++++++++++++++ include/uapi/linux/mtk-fd-v4l2-controls.h | 69 + include/uapi/linux/v4l2-controls.h | 4 + 8 files changed, 1468 insertions(+) create mode 100644 drivers/media/platform/mtk-isp/fd/Kconfig create mode 100644 drivers/media/platform/mtk-isp/fd/Makefile create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd.h create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd_40.c create mode 100644 include/uapi/linux/mtk-fd-v4l2-controls.h[snip]
Quoted 39 lines
Quoted 9 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +9
-0+static int mtk_fd_job_abort (struct mtk_fd_dev *fd) +{ +u32 ret; + + ret =wait_for_completion_timeout (&fd->fd_irq_done, +msecs_to_jiffies (MTK_FD_HW_TIMEOUT )); +/* Reset FD HW */ +if (!ret) { +struct ipi_message fd_ipi_msg;drivers/media/platform/mtk-isp/fd/mtk_fd_40.c line 362
Quoted 10 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +10
-0+ + fd_ipi_msg.cmd_id =MTK_FD_IPI_CMD_RESET ; +if (scp_ipi_send (fd->scp_pdev,SCP_IPI_FD_CMD , &fd_ipi_msg, +sizeof (fd_ipi_msg),MTK_FD_IPI_SEND_TIMEOUT )) +dev_err (fd->dev,"FD Reset HW error\n" ); +return -ETIMEDOUT ; + } +return 0 ; +} +Continue the discussion about job abort in RFC v2,
I think the
job_abortcallback inv4l2_m2m_ops()might be useful.ref:
https://elixir.bootlin.com/linux/v5.4-rc2/source/drivers/media/v4l2-core/v4l2-mem2mem.c#L398
https://elixir.bootlin.com/linux/v5.4-rc2/source/include/media/v4l2-mem2mem.h#L43in
drivers/media/v4l2-core/v4l2-mem2mem.c#398v4l2_m2m_cancel_job()
...if (m2m_ctx->job_flags & TRANS_RUNNING) { spin_unlock_irqrestore(&m2m_dev->job_spinlock, flags); if (m2m_dev->m2m_ops->job_abort) m2m_dev->m2m_ops->job_abort(m2m_ctx->priv); dprintk("m2m_ctx %p running, will wait to complete\n", m2m_ctx); wait_event(m2m_ctx->finished, !(m2m_ctx->job_flags & TRANS_RUNNING)); } ...Quoted 3 lines
If this operation is set, we might use the
v4l2_m2m_cancel_job()when suspend, and it will domtk_fd_job_abort()here.Quoted 8 lines
The expectation for .job_abort() is that signals the hardware to instantly abandon the current job. Do we have a way to tell the
firmware/hardwareto do so?Also, suspend must not abort the current job. Anything that was already running is expected to complete successfuly and further jobs should continue executing once the system resumes.
I appreciate your comments and Pi-Hsun's patch,
Ok, I see.
For FD40, we can't tell thefirmware/hardwareto instantly abandon the current job.
Therefore, for suspend, we stop sending further jobs to hardware and wait for the completion of the running job.
For resume, we continue sending jobs to hardware.Quoted 38 lines
[snip]
Quoted 27 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c line 1142
Quoted 12 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +12
-0+ +static int mtk_fd_suspend (struct device *dev) +{ +struct mtk_fd_dev *fd =dev_get_drvdata (dev); + +if (pm_runtime_suspended (dev)) +return 0 ; + +if (fd->fd_stream_count) +if (mtk_fd_job_abort (fd)) +mtk_fd_hw_job_finish (fd,VB2_BUF_STATE_ERROR ); +To avoid
mtk_fd_hw_job_finish()trigger the next job,
I suppose that we could usev4l2_m2m_cancel_jobinstead ofjob_abortandjob_finishhere./**
*v4l2_m2m_cancel_job()- cancel pending jobs for the context
* @m2m_ctx: m2m context with jobs to be canceled
*
* In case of streamoff or release called on any context,
* 1] If the context is currently running, then abort job will be called
* 2] If the context is queued, then the context will be removed from
* thejob_queue
*/Quoted 8 lines
or another way,
we may add a flag and implementmtk_fd_job_ready()that reads the flag if we suspend, we set the flag and dojob_abortandjob_finish, even if it try enqueue, it will still not really queue the job, until we reset the flag inmtk_fd_resume().how do you think?
Quoted 9 lines
As per my comment above, suspend must just pause the execution of the jobs. It must not cause any jobs to be skipped.
After analyzing the m2m framework and existing m2m drivers I realized that they currently provide no way to correctly handle suspend/resume. Pi-Hsun has been looking into fixing this in crrev.com/c/1878112 and we'll send it upstream as soon as we get something that should handle all the cases correctly.
Ok, thanks for the patches.
Quoted 31 lines
Quoted 19 lines
Quoted 19 lines
+ /* suspend FD HW */
+writel(0x0, fd->fd_base + MTK_FD_REG_OFFSET_INT_EN);
+writel(0x0, fd->fd_base + MTK_FD_REG_OFFSET_HW_ENABLE);
+clk_disable_unprepare(fd->fd_clk);
+dev_dbg(dev, "%s:disable clock\n", __func__);drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +14
-0+ +return 0 ; +} + +static int mtk_fd_resume (struct device *dev) +{ +struct mtk_fd_dev *fd =dev_get_drvdata (dev); +int ret; + +if (pm_runtime_suspended (dev)) +return 0 ; + + ret =clk_prepare_enable (fd->fd_clk); +if (ret <0 ) {Quoted 8 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c line 1169
Quoted 8 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +8
-0+dev_dbg (dev,"Failed to open fd clk:%d\n" , ret); +return ret; + } + +/* resume FD HW */ +writel (MTK_FD_SET_HW_ENABLE , fd->fd_base +MTK_FD_REG_OFFSET_HW_ENABLE ); +writel (0x1 , fd->fd_base +MTK_FD_REG_OFFSET_INT_EN ); +dev_dbg (dev,"%s:enable clock\n" , __func__);By the way, we need to kick the m2m framework here to schedule further jobs. Pi-Hsun's patch will also take care of this.
Quoted 9 lines
Quoted 4 lines
Ok, I see.
I would like to use Pi-Hsun's patch, otherwise I would need to callv4l2_m2m_try_run()here.Yes, please include Pi-Hsun's patch (with original author, sign-off + your sign-off added) at the beginning of the next version of your series.
Done.
Quoted 50 lines
Quoted 41 lines
Quoted 41 lines
[snip]
Quoted 27 lines
Quoted 11 lines
include/uapi/linux/mtk-fd-v4l2-controls.h new file +11
-0+/* Set the face angle and directions to be detected */ +#define V4L2_CID_MTK_FD_DETECT_POSE (V4L2_CID_USER_MTK_FD_BASE + 1) + +/* Set image widths for an input image to be scaled down for face detection */ +#define V4L2_CID_MTK_FD_SCALE_DOWN_IMG_WIDTH (V4L2_CID_USER_MTK_FD_BASE + 2) + +/* Set image heights for an input image to be scaled down for face detection */ +#define V4L2_CID_MTK_FD_SCALE_DOWN_IMG_HEIGHT (V4L2_CID_USER_MTK_FD_BASE + 3) + +/* Set the length of scale down size array */ +#define V4L2_CID_MTK_FD_SCALE_IMG_NUM (V4L2_CID_USER_MTK_FD_BASE + 4)include/uapi/linux/mtk-fd-v4l2-controls.h line 68
Quoted 10 lines
include/uapi/linux/mtk-fd-v4l2-controls.h new file +10
-0+ +/* Set the detection speed, usually reducing accuracy. */ +#define V4L2_CID_MTK_FD_DETECT_SPEED (V4L2_CID_USER_MTK_FD_BASE + 5) + +/* Select the detection model or algorithm to be used. */ +#define V4L2_CID_MTK_FD_DETECTION_MODEL (V4L2_CID_USER_MTK_FD_BASE + 6) + +/* We reserve 16 controls for this driver. */ +#define V4L2_CID_MTK_FD_MAX 16 +For these control IDs, I think the following should be remained as chip specific controls.
V4L2_CID_MTK_FD_SCALE_DOWN_IMG_WIDTH,V4L2_CID_MTK_FD_SCALE_DOWN_IMG_HEIGHTandV4L2_CID_MTK_FD_SCALE_IMG_NUMQuoted 7 lines
Hope there would be standardizing face detection api that cover the rest
controls:V4L2_CID_MTK_FD_DETECT_POSE,V4L2_CID_MTK_FD_DETECT_SPEEDandV4L2_CID_MTK_FD_DETECTION_MODELWould you have any suggestions on how to propose the standard face detection apis?
Given no follow up feedback from the community, I think we can keep them as driver-specific, but should make sure that they have some reasonable default values in case an application doesn't recognize them.
Quoted 5 lines
Quoted 3 lines
Best regards,
TomaszShould I keep the file "
mtk-fd-v4l2-controls.h" which defines the control ids under the folder "/include/uapi/linux"?We should define the CID base for the FD driver in
v4l2-controls.h, but the controls themselves should be only defined inside the driver.Quoted 6 lines
For example:
https://elixir.bootlin.com/linux/v5.4-rc8/source/include/uapi/linux/v4l2-controls.h#L178
https://elixir.bootlin.com/linux/v5.4-rc8/source/drivers/media/i2c/adv7180.c#L181Best regards,
Tomasz
Quoted 7 lines
Appreciate for providing the example,
Would it be fine for me to put the private CID in themtk_fd.h(which is similar to before...)or follow the example to define insidemtk_fd_40.c??The next version is almost ready, maybe I can send it when I ready in a few days?
Since mtk_fd_40.c is the only place the definitions from mtk_fd.h are used, I'd suggest just moving all the contents to the .c file.
Best regards,
Tomasz
Re: [RFC PATCH V3 3/3] platform: mtk-isp: Add Mediatek FD driver
Hi Tomasz,
Quoted 45 lines · Tomasz Figa
Quoted 43 lines · Jerry-ch Chen
Hi Tomasz,
Quoted 39 lines · Tomasz Figa
Quoted 37 lines · Jerry-ch Chen
Hi Tomasz,
Quoted 33 lines · Tomasz Figa
Quoted 30 lines · Jerry-ch Chen
Hi Tomasz,
Quoted 27 lines · Jerry-ch Chen
From: Jerry-ch Chen <[email protected]>
This patch adds the driver of Face Detection (FD) unit in Mediatek camera system, providing face detection function.
The mtk-isp directory will contain drivers for multiple IP blocks found in Mediatek ISP system. It will include ISP Pass 1 driver (CAM), sensor interface driver, DIP driver and face detection driver.
Signed-off-by Jerry-ch Chen <[email protected]>--- drivers/media/platform/Kconfig | 2 + drivers/media/platform/Makefile | 2 + drivers/media/platform/mtk-isp/fd/Kconfig | 19 + drivers/media/platform/mtk-isp/fd/Makefile | 5 + drivers/media/platform/mtk-isp/fd/mtk_fd.h | 148 ++ drivers/media/platform/mtk-isp/fd/mtk_fd_40.c | 1219 +++++++++++++++++ include/uapi/linux/mtk-fd-v4l2-controls.h | 69 + include/uapi/linux/v4l2-controls.h | 4 + 8 files changed, 1468 insertions(+) create mode 100644 drivers/media/platform/mtk-isp/fd/Kconfig create mode 100644 drivers/media/platform/mtk-isp/fd/Makefile create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd.h create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd_40.c create mode 100644 include/uapi/linux/mtk-fd-v4l2-controls.h[snip]
[snip]
Quoted 62 lines
Quoted 62 lines
Quoted 50 lines
Quoted 41 lines
Quoted 41 lines
[snip]
Quoted 27 lines
Quoted 11 lines
(quoted diff) +11
-0+/* Set the face angle and directions to be detected */ +#define V4L2_CID_MTK_FD_DETECT_POSE (V4L2_CID_USER_MTK_FD_BASE + 1) + +/* Set image widths for an input image to be scaled down for face detection */ +#define V4L2_CID_MTK_FD_SCALE_DOWN_IMG_WIDTH (V4L2_CID_USER_MTK_FD_BASE + 2) + +/* Set image heights for an input image to be scaled down for face detection */ +#define V4L2_CID_MTK_FD_SCALE_DOWN_IMG_HEIGHT (V4L2_CID_USER_MTK_FD_BASE + 3) + +/* Set the length of scale down size array */ +#define V4L2_CID_MTK_FD_SCALE_IMG_NUM (V4L2_CID_USER_MTK_FD_BASE + 4)Review
Quoted 10 lines
(quoted diff) +10
-0+ +/* Set the detection speed, usually reducing accuracy. */ +#define V4L2_CID_MTK_FD_DETECT_SPEED (V4L2_CID_USER_MTK_FD_BASE + 5) + +/* Select the detection model or algorithm to be used. */ +#define V4L2_CID_MTK_FD_DETECTION_MODEL (V4L2_CID_USER_MTK_FD_BASE + 6) + +/* We reserve 16 controls for this driver. */ +#define V4L2_CID_MTK_FD_MAX 16 +For these control IDs, I think the following should be remained as chip specific controls.
V4L2_CID_MTK_FD_SCALE_DOWN_IMG_WIDTH,V4L2_CID_MTK_FD_SCALE_DOWN_IMG_HEIGHTandV4L2_CID_MTK_FD_SCALE_IMG_NUMQuoted 7 lines
Hope there would be standardizing face detection api that cover the rest
controls:V4L2_CID_MTK_FD_DETECT_POSE,V4L2_CID_MTK_FD_DETECT_SPEEDandV4L2_CID_MTK_FD_DETECTION_MODELWould you have any suggestions on how to propose the standard face detection apis?
Given no follow up feedback from the community, I think we can keep them as driver-specific, but should make sure that they have some reasonable default values in case an application doesn't recognize them.
Quoted 5 lines
Quoted 3 lines
Best regards,
TomaszShould I keep the file "
mtk-fd-v4l2-controls.h" which defines the control ids under the folder "/include/uapi/linux"?We should define the CID base for the FD driver in
v4l2-controls.h, but the controls themselves should be only defined inside the driver.Quoted 6 lines
For example:
https://elixir.bootlin.com/linux/v5.4-rc8/source/include/uapi/linux/v4l2-controls.h#L178
https://elixir.bootlin.com/linux/v5.4-rc8/source/drivers/media/i2c/adv7180.c#L181Best regards,
TomaszAppreciate for providing the example,
Would it be fine for me to put the private CID in themtk_fd.h(which is similar to before...)or follow the example to define insidemtk_fd_40.c??
Quoted 8 lines
Quoted 2 lines
The next version is almost ready, maybe I can send it when I ready in a few days?
Since
mtk_fd_40.cis the only place the definitions frommtk_fd.hare used, I'd suggest just moving all the contents to the .c file.Best regards,
Tomasz
Done,
Thanks and best regards,
Jerry