Skip to contentSkip to conversation

media: platform: Add support for Face Detection (FD) on mt8183 SoC

RFC PATCH In review Jerry-ch Chen 5 participants 32 messages Dec 4, 2019 – Jan 6, 2021

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v4 cover media: platform: Add support for Face Detection (FD) on mt8183 SoC

Hello,

This RFC patch series is adding Face Detection (FD) driver on Mediatek mt8183 SoC. It belongs to the first Mediatek's camera driver series based on V4L2 and media controller framework. I posted the main part of the FD driver as RFC to discuss first and would like some review comments.

==============
Introduction
==============

Face Detection (FD) unit provides hardware accelerated face detection feature. It can detect different sizes of faces in a given image.

The driver is implemented as a normal V4L2 memory-to-memory device and supports V4L2 controls for detection settings. It has two buffer queues.

1. Video output buffer: RAW image for face detection.

2. Meta capture buffer: Result of the detected faces.

==================
Changes in v4
==================

RFC v4 includes the following modification:
1. Includes v4l2-mem2mem changes: add v4l2_m2m_suspend, v4l2_m2m_resume

2. Move FD V4L2 control ids back inside FD driver

3. Depend on newer SCP driver api

4. Add exit hw flow for FD driver

5. Add hardware timeout handling in the FD firmware

6. Move allocation of dma buffer from connect / disconnect to driver probe / remove

Todo:
- Add v4l2 control menus for private mtk fd control
==================
Changes in v3
==================

RFC v3 includes the following modification:
1. Adjust the private control ids and place them in include/uapi/linux/mtk-fd-v4l2-controls.h

2. Merge struct mtk_fd_hw info struct mtk_fd_dev

3. Define FD meta capture buffer in include/uapi/linux/videodev2.h

4. Remove the usage of get_reserved_memory by scp driver, use dma_alloc api instead

Todo:
- Add v4l2 control menus for private mtk fd control
- Refine the job finish flow when system suspend
==================
Changes in v2
==================

RFC v2 includes the following modification:
1. Implement FD as a V4L2 mem2mem driver

2. Replace meta input with V4L2 controls

==================
Changes in v1
==================

RFC v1 includes the following modification:
1. Uses Request API instead of FD's buffer collection design

2. removed unnecessary abstraction structurally, including mtk_fd_ctx and related ops

3. removed the fd_smem node from device tree

4. Fixed the common issues Tomasz commented on Mediatek ISP Pass 1's RFC v0 patch series

==================
Dependent patch
==================

FD driver depends on SCP driver. The patches are as following:

[1]. Add support for mt8183 SCP
https://patchwork.kernel.org/cover/1152350/

==================
Compliance test
==================

* Version: https://git.linuxtv.org/v4l-utils.git/commit/?id=b16f9e945d74aa5
* Note: Some failures are caused by the implementation of FD driver, whic is a m2m device with VIDEO_OUT and META_CAPTURE queues, therefore we can't set V4L2_CAP_VIDEO_M2M in device capability, and fail in some non-m2m v4l2 test cases.
* Test command: v4l2-compliance -m 2
* test output:

v4l2-compliance SHA: not available, 32 bits

Compliance test for mtk-fd-4.0 device /dev/media2:

Media Driver Info:

        Driver name      : mtk-fd-4.0
        Model            : mtk-fd-4.0
        Serial           :
        Bus info         : platform:1502b000.fd
        Media version    : 4.19.84
        Hardware revision: 0x00000000 (0)
        Driver version   : 4.19.84

Required ioctls:
test MEDIA_IOC_DEVICE_INFO: OK

Allow for multiple opens:
test second /dev/media2 open: OK
test MEDIA_IOC_DEVICE_INFO: OK
test for unlimited opens: OK

Media Controller ioctls:
test MEDIA_IOC_G_TOPOLOGY: OK
Entities: 3 Interfaces: 1 Pads: 4 Links: 4 test MEDIA_IOC_ENUM_ENTITIES/LINKS: OK test MEDIA_IOC_SETUP_LINK: OK

Total for mtk-fd-4.0 device /dev/media2: 7, Succeeded: 7, Failed: 0, Warnings: 0
--------------------------------------------------------------------------------
Compliance test for mtk-fd-4.0 device /dev/video32:

Driver Info:

        Driver name      : mtk-fd-4.0
        Card type        : mtk-fd-4.0
        Bus info         : platform:1502b000.fd
        Driver version   : 4.19.84
        Capabilities     : 0x84a02000

Video Output Multiplanar
Metadata Capture
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04a02000
Video Output Multiplanar
Metadata Capture
Streaming
Extended Pix Format
Media Driver Info:

        Driver name      : mtk-fd-4.0
        Model            : mtk-fd-4.0
        Serial           :
        Bus info         : platform:1502b000.fd
        Media version    : 4.19.84
        Hardware revision: 0x00000000 (0)
        Driver version   : 4.19.84

Interface Info:
ID : 0x0300000c
Type : V4L Video
Entity Info:

        ID               : 0x00000001 (1)
        Name             : mtk-fd-4.0-source
        Function         : V4L2 I/O
        Pad 0x01000002   : 0: Source
          Link 0x02000008: to remote pad 0x1000005 of entity 'mtk-fd-4.0-proc': Data, Enabled, Immutable

Required ioctls:
test MC information (see 'Media Driver Info' above): OK
fail: v4l2-compliance.cpp(668): dcaps & output_caps
test VIDIOC_QUERYCAP: FAIL

Allow for multiple opens:
test second /dev/video32 open: OK
fail: v4l2-compliance.cpp(668): dcaps & output_caps
test VIDIOC_QUERYCAP: FAIL
test VIDIOC_G/S_PRIORITY: OK
test for unlimited opens: OK

Debug ioctls:
test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported) test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported) test VIDIOC_ENUMAUDIO: OK (Not Supported)
test VIDIOC_G/S/ENUMINPUT: OK (Not Supported) test VIDIOC_G/S_AUDIO: OK (Not Supported)
Inputs: 0 Audio Inputs: 0 Tuners: 0

Output ioctls:
test VIDIOC_G/S_MODULATOR: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_ENUMAUDOUT: OK (Not Supported) test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported) test VIDIOC_G/S_AUDOUT: OK (Not Supported)
Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported) test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported) test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported) test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls:
test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK test VIDIOC_QUERYCTRL: OK
test VIDIOC_G/S_CTRL: OK
test VIDIOC_G/S/TRY_EXT_CTRLS: OK test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK test VIDIOC_G/S_JPEGCOMP: OK (Not Supported) Standard Controls: 1 Private Controls: 6

Format ioctls:
test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
test VIDIOC_G/S_PARM: OK (Not Supported)
test VIDIOC_G_FBUF: OK (Not Supported)
fail: v4l2-test-formats.cpp(457): pixelformat 00000000 () for buftype 10 not reported by ENUM_FMT
test VIDIOC_G_FMT: FAIL
fail: v4l2-test-formats.cpp(457): pixelformat 00000000 () for buftype 10 not reported by ENUM_FMT
test VIDIOC_TRY_FMT: FAIL
fail: v4l2-test-formats.cpp(457): pixelformat ffffffff (-BE) for buftype 10 not reported by ENUM_FMT
test VIDIOC_S_FMT: FAIL
test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
test Cropping: OK (Not Supported)
test Composing: OK (Not Supported)
test Scaling: OK (Not Supported)

Codec ioctls:
test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported) test VIDIOC_G_ENC_INDEX: OK (Not Supported) test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls:
fail: v4l2-test-buffers.cpp(667): q2.reqbufs(node->node2, 1) != EBUSY
test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: FAIL
test VIDIOC_EXPBUF: OK
test Requests: OK

Total for mtk-fd-4.0 device /dev/video32: 45, Succeeded: 39, Failed: 6, Warnings: 0

Grand Total for mtk-fd-4.0 device /dev/media2: 52, Succeeded: 46, Failed: 6, Warnings: 0

#1 lore raw

RFC PATCH v4 1/4 media: v4l2-mem2mem: add v4l2_m2m_suspend, v4l2_m2m_resume

From: Pi-Hsun Shih <[email protected]>

Add two functions that can be used to stop new jobs from being queued / continue running queued job. This can be used while a driver using m2m helper is going to suspend / wake up from resume, and can ensure that there's no job running in suspend process.

BUG=b:143046833
TEST=build

---
 drivers/media/v4l2-core/v4l2-mem2mem.c | 40 ++++++++++++++++++++++++++
 include/media/v4l2-mem2mem.h           | 22 ++++++++++++++
 2 files changed, 62 insertions(+)
drivers/media/v4l2-core/v4l2-mem2mem.c +40 -0
index 5bbdec55b7d7..76ba203e0035 100644
@@ -47,6 +47,10 @@ module_param(debug, bool, 0644);
 #define TRANS_ABORT		(1 << 2)
 
 
+/* The job queue is not running new jobs */
+#define QUEUE_PAUSED		(1 << 0)
+
+
 /* Offset base for buffers on the destination queue - used to distinguish
  * between source and destination buffers when mmapping - they receive the same
  * offsets but for different queues */
@@ -88,6 +92,7 @@ static const char * const m2m_entity_name[] = {
  * @job_queue:		instances queued to run
  * @job_spinlock:	protects job_queue
  * @job_work:		worker to run queued jobs.
+ * @job_queue_flags:	flags of the queue status, %QUEUE_PAUSED.
  * @m2m_ops:		driver callbacks
  */
 struct v4l2_m2m_dev {
@@ -105,6 +110,7 @@ struct v4l2_m2m_dev {
 	struct list_head	job_queue;
 	spinlock_t		job_spinlock;
 	struct work_struct	job_work;
+	unsigned long		job_queue_flags;
 
 	const struct v4l2_m2m_ops *m2m_ops;
 };
@@ -267,6 +273,12 @@ static void v4l2_m2m_try_run(struct v4l2_m2m_dev *m2m_dev)
 		return;
 	}
 
+	if (m2m_dev->job_queue_flags & QUEUE_PAUSED) {
+		spin_unlock_irqrestore(&m2m_dev->job_spinlock, flags);
+		dprintk("Running new jobs is paused\n");
+		return;
+	}
+
 	m2m_dev->curr_ctx = list_first_entry(&m2m_dev->job_queue,
 				   struct v4l2_m2m_ctx, queue);
 	m2m_dev->curr_ctx->job_flags |= TRANS_RUNNING;
@@ -447,6 +459,34 @@ void v4l2_m2m_job_finish(struct v4l2_m2m_dev *m2m_dev,
 }
 EXPORT_SYMBOL(v4l2_m2m_job_finish);
 
+void v4l2_m2m_suspend(struct v4l2_m2m_dev *m2m_dev)
+{
+	unsigned long flags;
+	struct v4l2_m2m_ctx *curr_ctx;
+
+	spin_lock_irqsave(&m2m_dev->job_spinlock, flags);
+	m2m_dev->job_queue_flags |= QUEUE_PAUSED;
+	curr_ctx = m2m_dev->curr_ctx;
+	spin_unlock_irqrestore(&m2m_dev->job_spinlock, flags);
+
+	if (curr_ctx)
+		wait_event(curr_ctx->finished,
+			   !(curr_ctx->job_flags & TRANS_RUNNING));
+}
+EXPORT_SYMBOL(v4l2_m2m_suspend);
+
+void v4l2_m2m_resume(struct v4l2_m2m_dev *m2m_dev)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&m2m_dev->job_spinlock, flags);
+	m2m_dev->job_queue_flags &= ~QUEUE_PAUSED;
+	spin_unlock_irqrestore(&m2m_dev->job_spinlock, flags);
+
+	v4l2_m2m_try_run(m2m_dev);
+}
+EXPORT_SYMBOL(v4l2_m2m_resume);
+
 int v4l2_m2m_reqbufs(struct file *file, struct v4l2_m2m_ctx *m2m_ctx,
 		     struct v4l2_requestbuffers *reqbufs)
 {
include/media/v4l2-mem2mem.h +22 -0
index 5467264771ec..119a195da390 100644
@@ -183,6 +183,28 @@ v4l2_m2m_buf_done(struct vb2_v4l2_buffer *buf, enum vb2_buffer_state state)
 	vb2_buffer_done(&buf->vb2_buf, state);
 }
 
+/**
+ * v4l2_m2m_suspend() - stop new jobs from being run and wait for current job
+ * to finish
+ *
+ * @m2m_dev: opaque pointer to the internal data to handle M2M context
+ *
+ * Called by a driver in the suspend hook. Stop new jobs from being run, and
+ * wait for current running job to finish.
+ */
+void v4l2_m2m_suspend(struct v4l2_m2m_dev *m2m_dev);
+
+/**
+ * v4l2_m2m_resume() - resume job running and try to run a queued job
+ *
+ * @m2m_dev: opaque pointer to the internal data to handle M2M context
+ *
+ * Called by a driver in the resume hook. This reverts the operation of
+ * v4l2_m2m_suspend() and allows job to be run. Also try to run a queued job if
+ * there is any.
+ */
+void v4l2_m2m_resume(struct v4l2_m2m_dev *m2m_dev);
+
 /**
  * v4l2_m2m_reqbufs() - multi-queue-aware REQBUFS multiplexer
  *
Signature · 1 line
2.18.0
Reviewed-by on v4 1/4
#2 lore raw

Re: [RFC PATCH V4 1/4] media: v4l2-mem2mem: add v4l2_m2m_suspend, v4l2_m2m_resume

Hi Jerry,

Quoted 131 lines · Jerry-ch Chen

From: Pi-Hsun Shih <[email protected]>

Add two functions that can be used to stop new jobs from being queued / continue running queued job. This can be used while a driver using m2m helper is going to suspend / wake up from resume, and can ensure that there's no job running in suspend process.

BUG=b:143046833
TEST=build

---
 drivers/media/v4l2-core/v4l2-mem2mem.c | 40 ++++++++++++++++++++++++++
 include/media/v4l2-mem2mem.h           | 22 ++++++++++++++
 2 files changed, 62 insertions(+)
drivers/media/v4l2-core/v4l2-mem2mem.c +40 -0
index 5bbdec55b7d7..76ba203e0035 100644
@@ -47,6 +47,10 @@ module_param(debug, bool, 0644);
 #define TRANS_ABORT		(1 << 2)
 
 
+/* The job queue is not running new jobs */
+#define QUEUE_PAUSED		(1 << 0)
+
+
 /* Offset base for buffers on the destination queue - used to distinguish
  * between source and destination buffers when mmapping - they receive the same
  * offsets but for different queues */
@@ -88,6 +92,7 @@ static const char * const m2m_entity_name[] = {
  * @job_queue:		instances queued to run
  * @job_spinlock:	protects job_queue
  * @job_work:		worker to run queued jobs.
+ * @job_queue_flags:	flags of the queue status, %QUEUE_PAUSED.
  * @m2m_ops:		driver callbacks
  */
 struct v4l2_m2m_dev {
@@ -105,6 +110,7 @@ struct v4l2_m2m_dev {
 	struct list_head	job_queue;
 	spinlock_t		job_spinlock;
 	struct work_struct	job_work;
+	unsigned long		job_queue_flags;
 
 	const struct v4l2_m2m_ops *m2m_ops;
 };
@@ -267,6 +273,12 @@ static void v4l2_m2m_try_run(struct v4l2_m2m_dev *m2m_dev)
 		return;
 	}
 
+	if (m2m_dev->job_queue_flags & QUEUE_PAUSED) {
+		spin_unlock_irqrestore(&m2m_dev->job_spinlock, flags);
+		dprintk("Running new jobs is paused\n");
+		return;
+	}
+
 	m2m_dev->curr_ctx = list_first_entry(&m2m_dev->job_queue,
 				   struct v4l2_m2m_ctx, queue);
 	m2m_dev->curr_ctx->job_flags |= TRANS_RUNNING;
@@ -447,6 +459,34 @@ void v4l2_m2m_job_finish(struct v4l2_m2m_dev *m2m_dev,
 }
 EXPORT_SYMBOL(v4l2_m2m_job_finish);
 
+void v4l2_m2m_suspend(struct v4l2_m2m_dev *m2m_dev)
+{
+	unsigned long flags;
+	struct v4l2_m2m_ctx *curr_ctx;
+
+	spin_lock_irqsave(&m2m_dev->job_spinlock, flags);
+	m2m_dev->job_queue_flags |= QUEUE_PAUSED;
+	curr_ctx = m2m_dev->curr_ctx;
+	spin_unlock_irqrestore(&m2m_dev->job_spinlock, flags);
+
+	if (curr_ctx)
+		wait_event(curr_ctx->finished,
+			   !(curr_ctx->job_flags & TRANS_RUNNING));
+}
+EXPORT_SYMBOL(v4l2_m2m_suspend);
+
+void v4l2_m2m_resume(struct v4l2_m2m_dev *m2m_dev)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&m2m_dev->job_spinlock, flags);
+	m2m_dev->job_queue_flags &= ~QUEUE_PAUSED;
+	spin_unlock_irqrestore(&m2m_dev->job_spinlock, flags);
+
+	v4l2_m2m_try_run(m2m_dev);
+}
+EXPORT_SYMBOL(v4l2_m2m_resume);
+
 int v4l2_m2m_reqbufs(struct file *file, struct v4l2_m2m_ctx *m2m_ctx,
 		     struct v4l2_requestbuffers *reqbufs)
 {
include/media/v4l2-mem2mem.h +22 -0
index 5467264771ec..119a195da390 100644
@@ -183,6 +183,28 @@ v4l2_m2m_buf_done(struct vb2_v4l2_buffer *buf, enum vb2_buffer_state state)
 	vb2_buffer_done(&buf->vb2_buf, state);
 }
 
+/**
+ * v4l2_m2m_suspend() - stop new jobs from being run and wait for current job
+ * to finish
+ *
+ * @m2m_dev: opaque pointer to the internal data to handle M2M context
+ *
+ * Called by a driver in the suspend hook. Stop new jobs from being run, and
+ * wait for current running job to finish.
+ */
+void v4l2_m2m_suspend(struct v4l2_m2m_dev *m2m_dev);
+
+/**
+ * v4l2_m2m_resume() - resume job running and try to run a queued job
+ *
+ * @m2m_dev: opaque pointer to the internal data to handle M2M context
+ *
+ * Called by a driver in the resume hook. This reverts the operation of
+ * v4l2_m2m_suspend() and allows job to be run. Also try to run a queued job if
+ * there is any.
+ */
+void v4l2_m2m_resume(struct v4l2_m2m_dev *m2m_dev);
+
 /**
  * v4l2_m2m_reqbufs() - multi-queue-aware REQBUFS multiplexer
  *
Signature · 1 line
2.18.0

[Corrected Hans's email address.]
Hans, does this look good to you?

Best regards,
Tomasz

#3 lore raw

Re: [RFC PATCH V4 1/4] media: v4l2-mem2mem: add v4l2_m2m_suspend, v4l2_m2m_resume

Hi Tomasz,

Quoted 128 lines · Tomasz Figa

Hi Jerry,

Quoted 125 lines · Jerry-ch Chen

From: Pi-Hsun Shih <[email protected]>

Add two functions that can be used to stop new jobs from being queued / continue running queued job. This can be used while a driver using m2m helper is going to suspend / wake up from resume, and can ensure that there's no job running in suspend process.

BUG=b:143046833
TEST=build

---
 drivers/media/v4l2-core/v4l2-mem2mem.c | 40 ++++++++++++++++++++++++++
 include/media/v4l2-mem2mem.h           | 22 ++++++++++++++
 2 files changed, 62 insertions(+)
drivers/media/v4l2-core/v4l2-mem2mem.c +40 -0
index 5bbdec55b7d7..76ba203e0035 100644
@@ -47,6 +47,10 @@ module_param(debug, bool, 0644);
 #define TRANS_ABORT		(1 << 2)
 
 
+/* The job queue is not running new jobs */
+#define QUEUE_PAUSED		(1 << 0)
+
+
 /* Offset base for buffers on the destination queue - used to distinguish
  * between source and destination buffers when mmapping - they receive the same
  * offsets but for different queues */
@@ -88,6 +92,7 @@ static const char * const m2m_entity_name[] = {
  * @job_queue:		instances queued to run
  * @job_spinlock:	protects job_queue
  * @job_work:		worker to run queued jobs.
+ * @job_queue_flags:	flags of the queue status, %QUEUE_PAUSED.
  * @m2m_ops:		driver callbacks
  */
 struct v4l2_m2m_dev {
@@ -105,6 +110,7 @@ struct v4l2_m2m_dev {
 	struct list_head	job_queue;
 	spinlock_t		job_spinlock;
 	struct work_struct	job_work;
+	unsigned long		job_queue_flags;
 
 	const struct v4l2_m2m_ops *m2m_ops;
 };
@@ -267,6 +273,12 @@ static void v4l2_m2m_try_run(struct v4l2_m2m_dev *m2m_dev)
 		return;
 	}
 
+	if (m2m_dev->job_queue_flags & QUEUE_PAUSED) {
+		spin_unlock_irqrestore(&m2m_dev->job_spinlock, flags);
+		dprintk("Running new jobs is paused\n");
+		return;
+	}
+
 	m2m_dev->curr_ctx = list_first_entry(&m2m_dev->job_queue,
 				   struct v4l2_m2m_ctx, queue);
 	m2m_dev->curr_ctx->job_flags |= TRANS_RUNNING;
@@ -447,6 +459,34 @@ void v4l2_m2m_job_finish(struct v4l2_m2m_dev *m2m_dev,
 }
 EXPORT_SYMBOL(v4l2_m2m_job_finish);
 
+void v4l2_m2m_suspend(struct v4l2_m2m_dev *m2m_dev)
+{
+	unsigned long flags;
+	struct v4l2_m2m_ctx *curr_ctx;
+
+	spin_lock_irqsave(&m2m_dev->job_spinlock, flags);
+	m2m_dev->job_queue_flags |= QUEUE_PAUSED;
+	curr_ctx = m2m_dev->curr_ctx;
+	spin_unlock_irqrestore(&m2m_dev->job_spinlock, flags);
+
+	if (curr_ctx)
+		wait_event(curr_ctx->finished,
+			   !(curr_ctx->job_flags & TRANS_RUNNING));
+}
+EXPORT_SYMBOL(v4l2_m2m_suspend);
+
+void v4l2_m2m_resume(struct v4l2_m2m_dev *m2m_dev)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&m2m_dev->job_spinlock, flags);
+	m2m_dev->job_queue_flags &= ~QUEUE_PAUSED;
+	spin_unlock_irqrestore(&m2m_dev->job_spinlock, flags);
+
+	v4l2_m2m_try_run(m2m_dev);
+}
+EXPORT_SYMBOL(v4l2_m2m_resume);
+
 int v4l2_m2m_reqbufs(struct file *file, struct v4l2_m2m_ctx *m2m_ctx,
 		     struct v4l2_requestbuffers *reqbufs)
 {
include/media/v4l2-mem2mem.h +21 -0
index 5467264771ec..119a195da390 100644
@@ -183,6 +183,28 @@ v4l2_m2m_buf_done(struct vb2_v4l2_buffer *buf, enum vb2_buffer_state state)
 	vb2_buffer_done(&buf->vb2_buf, state);
 }
 
+/**
+ * v4l2_m2m_suspend() - stop new jobs from being run and wait for current job
+ * to finish
+ *
+ * @m2m_dev: opaque pointer to the internal data to handle M2M context
+ *
+ * Called by a driver in the suspend hook. Stop new jobs from being run, and
+ * wait for current running job to finish.
+ */
+void v4l2_m2m_suspend(struct v4l2_m2m_dev *m2m_dev);
+
+/**
+ * v4l2_m2m_resume() - resume job running and try to run a queued job
+ *
+ * @m2m_dev: opaque pointer to the internal data to handle M2M context
+ *
+ * Called by a driver in the resume hook. This reverts the operation of
+ * v4l2_m2m_suspend() and allows job to be run. Also try to run a queued job if
+ * there is any.
+ */
+void v4l2_m2m_resume(struct v4l2_m2m_dev *m2m_dev);
Quoted 9 lines
Quoted 6 lines
include/media/v4l2-mem2mem.h +1 -0
+
 /**
  * v4l2_m2m_reqbufs() - multi-queue-aware REQBUFS multiplexer
  *
Signature · 1 line
2.18.0

Ok, I've added it in the commit message.

Thanks and Best regards,
Jerry

Quoted 6 lines

[Corrected Hans's email address.]
Hans, does this look good to you?

Best regards,
Tomasz

#4 lore raw

Re: [RFC PATCH V4 1/4] media: v4l2-mem2mem: add v4l2_m2m_suspend, v4l2_m2m_resume

In reply to Tomasz Figa
Quoted 137 lines · Tomasz Figa

Hi Jerry,

Quoted 131 lines · Jerry-ch Chen

From: Pi-Hsun Shih <[email protected]>

Add two functions that can be used to stop new jobs from being queued / continue running queued job. This can be used while a driver using m2m helper is going to suspend / wake up from resume, and can ensure that there's no job running in suspend process.

BUG=b:143046833
TEST=build

---
 drivers/media/v4l2-core/v4l2-mem2mem.c | 40 ++++++++++++++++++++++++++
 include/media/v4l2-mem2mem.h           | 22 ++++++++++++++
 2 files changed, 62 insertions(+)
drivers/media/v4l2-core/v4l2-mem2mem.c +40 -0
index 5bbdec55b7d7..76ba203e0035 100644
@@ -47,6 +47,10 @@ module_param(debug, bool, 0644);
 #define TRANS_ABORT		(1 << 2)
 
 
+/* The job queue is not running new jobs */
+#define QUEUE_PAUSED		(1 << 0)
+
+
 /* Offset base for buffers on the destination queue - used to distinguish
  * between source and destination buffers when mmapping - they receive the same
  * offsets but for different queues */
@@ -88,6 +92,7 @@ static const char * const m2m_entity_name[] = {
  * @job_queue:		instances queued to run
  * @job_spinlock:	protects job_queue
  * @job_work:		worker to run queued jobs.
+ * @job_queue_flags:	flags of the queue status, %QUEUE_PAUSED.
  * @m2m_ops:		driver callbacks
  */
 struct v4l2_m2m_dev {
@@ -105,6 +110,7 @@ struct v4l2_m2m_dev {
 	struct list_head	job_queue;
 	spinlock_t		job_spinlock;
 	struct work_struct	job_work;
+	unsigned long		job_queue_flags;
 
 	const struct v4l2_m2m_ops *m2m_ops;
 };
@@ -267,6 +273,12 @@ static void v4l2_m2m_try_run(struct v4l2_m2m_dev *m2m_dev)
 		return;
 	}
 
+	if (m2m_dev->job_queue_flags & QUEUE_PAUSED) {
+		spin_unlock_irqrestore(&m2m_dev->job_spinlock, flags);
+		dprintk("Running new jobs is paused\n");
+		return;
+	}
+
 	m2m_dev->curr_ctx = list_first_entry(&m2m_dev->job_queue,
 				   struct v4l2_m2m_ctx, queue);
 	m2m_dev->curr_ctx->job_flags |= TRANS_RUNNING;
@@ -447,6 +459,34 @@ void v4l2_m2m_job_finish(struct v4l2_m2m_dev *m2m_dev,
 }
 EXPORT_SYMBOL(v4l2_m2m_job_finish);
 
+void v4l2_m2m_suspend(struct v4l2_m2m_dev *m2m_dev)
+{
+	unsigned long flags;
+	struct v4l2_m2m_ctx *curr_ctx;
+
+	spin_lock_irqsave(&m2m_dev->job_spinlock, flags);
+	m2m_dev->job_queue_flags |= QUEUE_PAUSED;
+	curr_ctx = m2m_dev->curr_ctx;
+	spin_unlock_irqrestore(&m2m_dev->job_spinlock, flags);
+
+	if (curr_ctx)
+		wait_event(curr_ctx->finished,
+			   !(curr_ctx->job_flags & TRANS_RUNNING));
+}
+EXPORT_SYMBOL(v4l2_m2m_suspend);
+
+void v4l2_m2m_resume(struct v4l2_m2m_dev *m2m_dev)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&m2m_dev->job_spinlock, flags);
+	m2m_dev->job_queue_flags &= ~QUEUE_PAUSED;
+	spin_unlock_irqrestore(&m2m_dev->job_spinlock, flags);
+
+	v4l2_m2m_try_run(m2m_dev);
+}
+EXPORT_SYMBOL(v4l2_m2m_resume);
+
 int v4l2_m2m_reqbufs(struct file *file, struct v4l2_m2m_ctx *m2m_ctx,
 		     struct v4l2_requestbuffers *reqbufs)
 {
include/media/v4l2-mem2mem.h +22 -0
index 5467264771ec..119a195da390 100644
@@ -183,6 +183,28 @@ v4l2_m2m_buf_done(struct vb2_v4l2_buffer *buf, enum vb2_buffer_state state)
 	vb2_buffer_done(&buf->vb2_buf, state);
 }
 
+/**
+ * v4l2_m2m_suspend() - stop new jobs from being run and wait for current job
+ * to finish
+ *
+ * @m2m_dev: opaque pointer to the internal data to handle M2M context
+ *
+ * Called by a driver in the suspend hook. Stop new jobs from being run, and
+ * wait for current running job to finish.
+ */
+void v4l2_m2m_suspend(struct v4l2_m2m_dev *m2m_dev);
+
+/**
+ * v4l2_m2m_resume() - resume job running and try to run a queued job
+ *
+ * @m2m_dev: opaque pointer to the internal data to handle M2M context
+ *
+ * Called by a driver in the resume hook. This reverts the operation of
+ * v4l2_m2m_suspend() and allows job to be run. Also try to run a queued job if
+ * there is any.
+ */
+void v4l2_m2m_resume(struct v4l2_m2m_dev *m2m_dev);
+
 /**
  * v4l2_m2m_reqbufs() - multi-queue-aware REQBUFS multiplexer
  *
Signature · 1 line
2.18.0
Quoted 2 lines

[Corrected Hans's email address.]
Hans, does this look good to you?

Yes, this looks good.

Sorry for the late reply.

I assume this will be part of a future patch series that calls these new functions?

Regards,

Hans

#5 lore raw

Re: [RFC PATCH V4 1/4] media: v4l2-mem2mem: add v4l2_m2m_suspend, v4l2_m2m_resume

Quoted 139 lines · Hans Verkuil
Quoted 137 lines · Tomasz Figa

Hi Jerry,

Quoted 130 lines · Jerry-ch Chen

From: Pi-Hsun Shih <[email protected]>

Add two functions that can be used to stop new jobs from being queued / continue running queued job. This can be used while a driver using m2m helper is going to suspend / wake up from resume, and can ensure that there's no job running in suspend process.

BUG=b:143046833
TEST=build

---
 drivers/media/v4l2-core/v4l2-mem2mem.c | 40 ++++++++++++++++++++++++++
 include/media/v4l2-mem2mem.h           | 22 ++++++++++++++
 2 files changed, 62 insertions(+)
drivers/media/v4l2-core/v4l2-mem2mem.c +40 -0
index 5bbdec55b7d7..76ba203e0035 100644
@@ -47,6 +47,10 @@ module_param(debug, bool, 0644);
 #define TRANS_ABORT         (1 << 2)
 
 
+/* The job queue is not running new jobs */
+#define QUEUE_PAUSED                (1 << 0)
+
+
 /* Offset base for buffers on the destination queue - used to distinguish
  * between source and destination buffers when mmapping - they receive the same
  * offsets but for different queues */
@@ -88,6 +92,7 @@ static const char * const m2m_entity_name[] = {
  * @job_queue:              instances queued to run
  * @job_spinlock:   protects job_queue
  * @job_work:               worker to run queued jobs.
+ * @job_queue_flags:        flags of the queue status, %QUEUE_PAUSED.
  * @m2m_ops:                driver callbacks
  */
 struct v4l2_m2m_dev {
@@ -105,6 +110,7 @@ struct v4l2_m2m_dev {
     struct list_head        job_queue;
     spinlock_t              job_spinlock;
     struct work_struct      job_work;
+    unsigned long           job_queue_flags;
 
     const struct v4l2_m2m_ops *m2m_ops;
 };
@@ -267,6 +273,12 @@ static void v4l2_m2m_try_run(struct v4l2_m2m_dev *m2m_dev)
             return;
     }
 
+    if (m2m_dev->job_queue_flags & QUEUE_PAUSED) {
+            spin_unlock_irqrestore(&m2m_dev->job_spinlock, flags);
+            dprintk("Running new jobs is paused\n");
+            return;
+    }
+
     m2m_dev->curr_ctx = list_first_entry(&m2m_dev->job_queue,
                                struct v4l2_m2m_ctx, queue);
     m2m_dev->curr_ctx->job_flags |= TRANS_RUNNING;
@@ -447,6 +459,34 @@ void v4l2_m2m_job_finish(struct v4l2_m2m_dev *m2m_dev,
 }
 EXPORT_SYMBOL(v4l2_m2m_job_finish);
 
+void v4l2_m2m_suspend(struct v4l2_m2m_dev *m2m_dev)
+{
+    unsigned long flags;
+    struct v4l2_m2m_ctx *curr_ctx;
+
+    spin_lock_irqsave(&m2m_dev->job_spinlock, flags);
+    m2m_dev->job_queue_flags |= QUEUE_PAUSED;
+    curr_ctx = m2m_dev->curr_ctx;
+    spin_unlock_irqrestore(&m2m_dev->job_spinlock, flags);
+
+    if (curr_ctx)
+            wait_event(curr_ctx->finished,
+                       !(curr_ctx->job_flags & TRANS_RUNNING));
+}
+EXPORT_SYMBOL(v4l2_m2m_suspend);
+
+void v4l2_m2m_resume(struct v4l2_m2m_dev *m2m_dev)
+{
+    unsigned long flags;
+
+    spin_lock_irqsave(&m2m_dev->job_spinlock, flags);
+    m2m_dev->job_queue_flags &= ~QUEUE_PAUSED;
+    spin_unlock_irqrestore(&m2m_dev->job_spinlock, flags);
+
+    v4l2_m2m_try_run(m2m_dev);
+}
+EXPORT_SYMBOL(v4l2_m2m_resume);
+
 int v4l2_m2m_reqbufs(struct file *file, struct v4l2_m2m_ctx *m2m_ctx,
                  struct v4l2_requestbuffers *reqbufs)
 {
include/media/v4l2-mem2mem.h +22 -1
index 5467264771ec..119a195da390 100644
@@ -183,6 +183,28 @@ v4l2_m2m_buf_done(struct vb2_v4l2_buffer *buf, enum vb2_buffer_state state)
     vb2_buffer_done(&buf->vb2_buf, state);
 }
 
+/**
+ * v4l2_m2m_suspend() - stop new jobs from being run and wait for current job
+ * to finish
+ *
+ * @m2m_dev: opaque pointer to the internal data to handle M2M context
+ *
+ * Called by a driver in the suspend hook. Stop new jobs from being run, and
+ * wait for current running job to finish.
+ */
+void v4l2_m2m_suspend(struct v4l2_m2m_dev *m2m_dev);
+
+/**
+ * v4l2_m2m_resume() - resume job running and try to run a queued job
+ *
+ * @m2m_dev: opaque pointer to the internal data to handle M2M context
+ *
+ * Called by a driver in the resume hook. This reverts the operation of
+ * v4l2_m2m_suspend() and allows job to be run. Also try to run a queued job if
+ * there is any.
+ */
+void v4l2_m2m_resume(struct v4l2_m2m_dev *m2m_dev);
+
 /**
  * v4l2_m2m_reqbufs() - multi-queue-aware REQBUFS multiplexer
  *
--
Quoted 1 line

2.18.0

Quoted 6 lines
Quoted 2 lines

[Corrected Hans's email address.]
Hans, does this look good to you?

Yes, this looks good.

Sorry for the late reply.

No worries! Thanks a lot.

Quoted 2 lines

I assume this will be part of a future patch series that calls these new functions?

The mtk-jpeg encoder series depends on this patch as well, so I guess it would go together with whichever is ready first.

I would also envision someone changing the other existing drivers to use the helpers, as I'm pretty much sure some of them don't handle suspend/resume correctly.

Best regards,
Tomasz

#6 lore raw

Re: [RFC PATCH V4 1/4] media: v4l2-mem2mem: add v4l2_m2m_suspend, v4l2_m2m_resume

Hi everyone,

Thanks for the patch.

Quoted 9 lines · Tomasz Figa
Quoted 7 lines · Hans Verkuil
Quoted 5 lines · Tomasz Figa

Hi Jerry,

Quoted 2 lines · Jerry-ch Chen

From: Pi-Hsun Shih <[email protected]>

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Add two functions that can be used to stop new jobs from being queued / continue running queued job. This can be used while a driver using m2m helper is going to suspend / wake up from resume, and can ensure that there's no job running in suspend process.

BUG=b:143046833
TEST=build

BUG/TEST tags need to be removed.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

This mail address probably needs correction?

Quoted 132 lines
Quoted 122 lines
Quoted 122 lines
Quoted 118 lines
---
 drivers/media/v4l2-core/v4l2-mem2mem.c | 40 ++++++++++++++++++++++++++
 include/media/v4l2-mem2mem.h           | 22 ++++++++++++++
 2 files changed, 62 insertions(+)
drivers/media/v4l2-core/v4l2-mem2mem.c +40 -0
index 5bbdec55b7d7..76ba203e0035 100644
@@ -47,6 +47,10 @@ module_param(debug, bool, 0644);
 #define TRANS_ABORT         (1 << 2)
 
 
+/* The job queue is not running new jobs */
+#define QUEUE_PAUSED                (1 << 0)
+
+
 /* Offset base for buffers on the destination queue - used to distinguish
  * between source and destination buffers when mmapping - they receive the same
  * offsets but for different queues */
@@ -88,6 +92,7 @@ static const char * const m2m_entity_name[] = {
  * @job_queue:              instances queued to run
  * @job_spinlock:   protects job_queue
  * @job_work:               worker to run queued jobs.
+ * @job_queue_flags:        flags of the queue status, %QUEUE_PAUSED.
  * @m2m_ops:                driver callbacks
  */
 struct v4l2_m2m_dev {
@@ -105,6 +110,7 @@ struct v4l2_m2m_dev {
     struct list_head        job_queue;
     spinlock_t              job_spinlock;
     struct work_struct      job_work;
+    unsigned long           job_queue_flags;
 
     const struct v4l2_m2m_ops *m2m_ops;
 };
@@ -267,6 +273,12 @@ static void v4l2_m2m_try_run(struct v4l2_m2m_dev *m2m_dev)
             return;
     }
 
+    if (m2m_dev->job_queue_flags & QUEUE_PAUSED) {
+            spin_unlock_irqrestore(&m2m_dev->job_spinlock, flags);
+            dprintk("Running new jobs is paused\n");
+            return;
+    }
+
     m2m_dev->curr_ctx = list_first_entry(&m2m_dev->job_queue,
                                struct v4l2_m2m_ctx, queue);
     m2m_dev->curr_ctx->job_flags |= TRANS_RUNNING;
@@ -447,6 +459,34 @@ void v4l2_m2m_job_finish(struct v4l2_m2m_dev *m2m_dev,
 }
 EXPORT_SYMBOL(v4l2_m2m_job_finish);
 
+void v4l2_m2m_suspend(struct v4l2_m2m_dev *m2m_dev)
+{
+    unsigned long flags;
+    struct v4l2_m2m_ctx *curr_ctx;
+
+    spin_lock_irqsave(&m2m_dev->job_spinlock, flags);
+    m2m_dev->job_queue_flags |= QUEUE_PAUSED;
+    curr_ctx = m2m_dev->curr_ctx;
+    spin_unlock_irqrestore(&m2m_dev->job_spinlock, flags);
+
+    if (curr_ctx)
+            wait_event(curr_ctx->finished,
+                       !(curr_ctx->job_flags & TRANS_RUNNING));
+}
+EXPORT_SYMBOL(v4l2_m2m_suspend);
+
+void v4l2_m2m_resume(struct v4l2_m2m_dev *m2m_dev)
+{
+    unsigned long flags;
+
+    spin_lock_irqsave(&m2m_dev->job_spinlock, flags);
+    m2m_dev->job_queue_flags &= ~QUEUE_PAUSED;
+    spin_unlock_irqrestore(&m2m_dev->job_spinlock, flags);
+
+    v4l2_m2m_try_run(m2m_dev);
+}
+EXPORT_SYMBOL(v4l2_m2m_resume);
+
 int v4l2_m2m_reqbufs(struct file *file, struct v4l2_m2m_ctx *m2m_ctx,
                  struct v4l2_requestbuffers *reqbufs)
 {
include/media/v4l2-mem2mem.h +22 -1
index 5467264771ec..119a195da390 100644
@@ -183,6 +183,28 @@ v4l2_m2m_buf_done(struct vb2_v4l2_buffer *buf, enum vb2_buffer_state state)
     vb2_buffer_done(&buf->vb2_buf, state);
 }
 
+/**
+ * v4l2_m2m_suspend() - stop new jobs from being run and wait for current job
+ * to finish
+ *
+ * @m2m_dev: opaque pointer to the internal data to handle M2M context
+ *
+ * Called by a driver in the suspend hook. Stop new jobs from being run, and
+ * wait for current running job to finish.
+ */
+void v4l2_m2m_suspend(struct v4l2_m2m_dev *m2m_dev);
+
+/**
+ * v4l2_m2m_resume() - resume job running and try to run a queued job
+ *
+ * @m2m_dev: opaque pointer to the internal data to handle M2M context
+ *
+ * Called by a driver in the resume hook. This reverts the operation of
+ * v4l2_m2m_suspend() and allows job to be run. Also try to run a queued job if
+ * there is any.
+ */
+void v4l2_m2m_resume(struct v4l2_m2m_dev *m2m_dev);
+
 /**
  * v4l2_m2m_reqbufs() - multi-queue-aware REQBUFS multiplexer
  *
--
Quoted 1 line

2.18.0

Quoted 6 lines
Quoted 2 lines

[Corrected Hans's email address.]
Hans, does this look good to you?

Yes, this looks good.

Sorry for the late reply.

No worries! Thanks a lot.

Quoted 1 line
Quoted 9 lines
Quoted 1 line

I assume this will be part of a future patch series that calls these new functions?

The mtk-jpeg encoder series depends on this patch as well, so I guess it would go together with whichever is ready first.

I would also envision someone changing the other existing drivers to use the helpers, as I'm pretty much sure some of them don't handle suspend/resume correctly.

This indeed looks very good. If I understood the issue properly, the change would be useful for both stateless (e.g. hantro, et al) and stateful (e.g. coda) codecs.

Hantro uses pm_runtime_force_suspend, and I believe that could is enough for proper suspend/resume operation.

I'm not seeing any code in CODA to handle this, so not sure how it's handling suspend/resume.

Maybe we can have CODA as the first user, given it's a well-maintained driver and should be fairly easy to test.

Regards,
Ezequiel

#7 lore raw

Re: [RFC PATCH V4 1/4] media: v4l2-mem2mem: add v4l2_m2m_suspend, v4l2_m2m_resume

Quoted 18 lines · Ezequiel Garcia

Hi everyone,

Thanks for the patch.

Quoted 13 lines · Tomasz Figa
Quoted 11 lines · Hans Verkuil
Quoted 9 lines · Tomasz Figa

Hi Jerry,

Quoted 6 lines · Jerry-ch Chen

From: Pi-Hsun Shih <[email protected]>

Add two functions that can be used to stop new jobs from being queued / continue running queued job. This can be used while a driver using m2m helper is going to suspend / wake up from resume, and can ensure that there's no job running in suspend process.

[snip]

Quoted 11 lines
Quoted 10 lines
Quoted 2 lines

I assume this will be part of a future patch series that calls these new functions?

The mtk-jpeg encoder series depends on this patch as well, so I guess it would go together with whichever is ready first.

I would also envision someone changing the other existing drivers to use the helpers, as I'm pretty much sure some of them don't handle suspend/resume correctly.

Quoted 6 lines

This indeed looks very good. If I understood the issue properly, the change would be useful for both stateless (e.g. hantro, et al) and stateful (e.g. coda) codecs.

Hantro uses pm_runtime_force_suspend, and I believe that could is enough for proper suspend/resume operation.

Unfortunately, no. :(

If the decoder is already decoding a frame, that would forcefully power off the hardware and possibly even cause a system lockup if we are unlucky to gate a clock in the middle of a bus transaction.

I just inspected the code now and actually found one more bug in its power management handling. device_run() calls clk_bulk_enable() before pm_runtime_get_sync(), but only the latter is guaranteed to actually power on the relevant power domains, so we end up clocking unpowered hardware.

Quoted 6 lines

I'm not seeing any code in CODA to handle this, so not sure how it's handling suspend/resume.

Maybe we can have CODA as the first user, given it's a well-maintained driver and should be fairly easy to test.

I remember checking a number of drivers using the m2m helpers randomly and none of them implemented suspend/resume correctly. I suppose that was not discovered because normally the userspace itself would stop the operation before the system is suspended, although it's not an API guarantee.

Best regards,
Tomasz

#8 lore raw

Re: [RFC PATCH V4 1/4] media: v4l2-mem2mem: add v4l2_m2m_suspend, v4l2_m2m_resume

Quoted 36 lines · Tomasz Figa
Quoted 18 lines · Ezequiel Garcia

Hi everyone,

Thanks for the patch.

Quoted 13 lines · Tomasz Figa
Quoted 11 lines · Hans Verkuil
Quoted 9 lines · Tomasz Figa

Hi Jerry,

Quoted 6 lines · Jerry-ch Chen

From: Pi-Hsun Shih <[email protected]>

Add two functions that can be used to stop new jobs from being queued / continue running queued job. This can be used while a driver using m2m helper is going to suspend / wake up from resume, and can ensure that there's no job running in suspend process.

[snip]

Quoted 15 lines
Quoted 10 lines
Quoted 2 lines

I assume this will be part of a future patch series that calls these new functions?

The mtk-jpeg encoder series depends on this patch as well, so I guess it would go together with whichever is ready first.

I would also envision someone changing the other existing drivers to use the helpers, as I'm pretty much sure some of them don't handle suspend/resume correctly.

This indeed looks very good. If I understood the issue properly, the change would be useful for both stateless (e.g. hantro, et al) and stateful (e.g. coda) codecs.

Quoted 9 lines
Quoted 2 lines

Hantro uses pm_runtime_force_suspend, and I believe that could is enough for proper suspend/resume operation.

Unfortunately, no. :(

If the decoder is already decoding a frame, that would forcefully power off the hardware and possibly even cause a system lockup if we are unlucky to gate a clock in the middle of a bus transaction.

pm_runtime_force_suspend calls pm_runtime_disable, which says:

"""
Increment power.disable_depth for the device and if it was zero previously, cancel all pending runtime PM requests for the device and wait for all operations in progress to complete.
"""

Doesn't this mean it waits for the current job (if there is one) and prevents any new jobs to be issued?

Quoted 6 lines

I just inspected the code now and actually found one more bug in its power management handling. device_run() calls clk_bulk_enable() before pm_runtime_get_sync(), but only the latter is guaranteed to actually power on the relevant power domains, so we end up clocking unpowered hardware.

How about we just move clk_enable/disable to runtime PM?

Since we use autosuspend delay, it theoretically has some impact, which is why I was refraining from doing so.

I can't decide if this impact would be marginal or significant.

Quoted 4 lines
Quoted 4 lines

I'm not seeing any code in CODA to handle this, so not sure how it's handling suspend/resume.

Quoted 9 lines
Quoted 2 lines

Maybe we can have CODA as the first user, given it's a well-maintained driver and should be fairly easy to test.

I remember checking a number of drivers using the m2m helpers randomly and none of them implemented suspend/resume correctly. I suppose that was not discovered because normally the userspace itself would stop the operation before the system is suspended, although it's not an API guarantee.

Indeed. Do you have any recomendations for how we could test this case to make sure we are handling it correctly?

Quoted 2 lines

Best regards,
Tomasz

#9 lore raw

Re: [RFC PATCH V4 1/4] media: v4l2-mem2mem: add v4l2_m2m_suspend, v4l2_m2m_resume

Quoted 51 lines · Ezequiel Garcia
Quoted 36 lines · Tomasz Figa
Quoted 18 lines · Ezequiel Garcia

Hi everyone,

Thanks for the patch.

Quoted 13 lines · Tomasz Figa
Quoted 11 lines · Hans Verkuil
Quoted 9 lines · Tomasz Figa

Hi Jerry,

Quoted 6 lines · Jerry-ch Chen

From: Pi-Hsun Shih <[email protected]>

Add two functions that can be used to stop new jobs from being queued / continue running queued job. This can be used while a driver using m2m helper is going to suspend / wake up from resume, and can ensure that there's no job running in suspend process.

[snip]

Quoted 15 lines
Quoted 10 lines
Quoted 2 lines

I assume this will be part of a future patch series that calls these new functions?

The mtk-jpeg encoder series depends on this patch as well, so I guess it would go together with whichever is ready first.

I would also envision someone changing the other existing drivers to use the helpers, as I'm pretty much sure some of them don't handle suspend/resume correctly.

This indeed looks very good. If I understood the issue properly, the change would be useful for both stateless (e.g. hantro, et al) and stateful (e.g. coda) codecs.

Quoted 9 lines
Quoted 2 lines

Hantro uses pm_runtime_force_suspend, and I believe that could is enough for proper suspend/resume operation.

Unfortunately, no. :(

If the decoder is already decoding a frame, that would forcefully power off the hardware and possibly even cause a system lockup if we are unlucky to gate a clock in the middle of a bus transaction.

pm_runtime_force_suspend calls pm_runtime_disable, which says:

Quoted 9 lines

"""
Increment power.disable_depth for the device and if it was zero previously, cancel all pending runtime PM requests for the device and wait for all operations in progress to complete.
"""

Doesn't this mean it waits for the current job (if there is one) and prevents any new jobs to be issued?

I'd love if the PM runtime subsystem handled job management of all the driver subsystems automatically, but at the moment it's not aware of any jobs. :) The description says as much as it says - it stops any internal jobs of the PM subsystem - i.e. asynchronous suspend/resume requests. It doesn't have any awareness of V4L2 M2M jobs.

Quoted 7 lines
Quoted 6 lines

I just inspected the code now and actually found one more bug in its power management handling. device_run() calls clk_bulk_enable() before pm_runtime_get_sync(), but only the latter is guaranteed to actually power on the relevant power domains, so we end up clocking unpowered hardware.

Quoted 7 lines

How about we just move clk_enable/disable to runtime PM?

Since we use autosuspend delay, it theoretically has some impact, which is why I was refraining from doing so.

I can't decide if this impact would be marginal or significant.

I'd also refrain from doing this. Clock gating corresponds to the bigger part of the power savings from runtime power management, since it stops the dynamic power consumption and only leaves the static leakage. That said, the Hantro IP blocks have some internal clock gating as well, so it might not be as pronounced, depending on the custom vendor integration logic surrounding the Hantro hardware.

Actually even if autosuspend is not used, the runtime PM subsystem has some internal back-off mechanism based on measured power on and power off latencies. The driver should call pm_runtime_get_sync() first and then enable any necessary clocks. I can see that currently inside the resume callback we have some hardware accesses. If those really need to be there, they should be surrounded with appropriate clock enable and clock disable calls.

Quoted 14 lines
Quoted 4 lines
Quoted 4 lines

I'm not seeing any code in CODA to handle this, so not sure how it's handling suspend/resume.

Quoted 9 lines
Quoted 2 lines

Maybe we can have CODA as the first user, given it's a well-maintained driver and should be fairly easy to test.

I remember checking a number of drivers using the m2m helpers randomly and none of them implemented suspend/resume correctly. I suppose that was not discovered because normally the userspace itself would stop the operation before the system is suspended, although it's not an API guarantee.

Quoted 2 lines

Indeed. Do you have any recomendations for how we could test this case to make sure we are handling it correctly?

I'd say that a simple offscreen command line gstreamer/ffmpeg decode with suspend/resume loop in another session should be able to trigger some issues.

Best regards,
Tomasz

#10 lore raw

Re: [RFC PATCH V4 1/4] media: v4l2-mem2mem: add v4l2_m2m_suspend, v4l2_m2m_resume

Quoted 64 lines · Tomasz Figa
Quoted 51 lines · Ezequiel Garcia
Quoted 36 lines · Tomasz Figa
Quoted 18 lines · Ezequiel Garcia

Hi everyone,

Thanks for the patch.

Quoted 13 lines · Tomasz Figa
Quoted 11 lines · Hans Verkuil
Quoted 9 lines · Tomasz Figa

Hi Jerry,

Quoted 6 lines · Jerry-ch Chen

From: Pi-Hsun Shih <[email protected]>

Add two functions that can be used to stop new jobs from being queued / continue running queued job. This can be used while a driver using m2m helper is going to suspend / wake up from resume, and can ensure that there's no job running in suspend process.

[snip]

Quoted 15 lines
Quoted 10 lines
Quoted 2 lines

I assume this will be part of a future patch series that calls these new functions?

The mtk-jpeg encoder series depends on this patch as well, so I guess it would go together with whichever is ready first.

I would also envision someone changing the other existing drivers to use the helpers, as I'm pretty much sure some of them don't handle suspend/resume correctly.

This indeed looks very good. If I understood the issue properly, the change would be useful for both stateless (e.g. hantro, et al) and stateful (e.g. coda) codecs.

Quoted 9 lines
Quoted 2 lines

Hantro uses pm_runtime_force_suspend, and I believe that could is enough for proper suspend/resume operation.

Unfortunately, no. :(

If the decoder is already decoding a frame, that would forcefully power off the hardware and possibly even cause a system lockup if we are unlucky to gate a clock in the middle of a bus transaction.

pm_runtime_force_suspend calls pm_runtime_disable, which says:

Quoted 9 lines

"""
Increment power.disable_depth for the device and if it was zero previously, cancel all pending runtime PM requests for the device and wait for all operations in progress to complete.
"""

Doesn't this mean it waits for the current job (if there is one) and prevents any new jobs to be issued?

Quoted 6 lines

I'd love if the PM runtime subsystem handled job management of all the driver subsystems automatically, but at the moment it's not aware of any jobs. :) The description says as much as it says - it stops any internal jobs of the PM subsystem - i.e. asynchronous suspend/resume requests. It doesn't have any awareness of V4L2 M2M jobs.

Doh, of course. I saw "pending requests" and somehow imagined it would wait for the runtime_put.

I see now that these patches are the way to go.

Quoted 15 lines
Quoted 7 lines
Quoted 6 lines

I just inspected the code now and actually found one more bug in its power management handling. device_run() calls clk_bulk_enable() before pm_runtime_get_sync(), but only the latter is guaranteed to actually power on the relevant power domains, so we end up clocking unpowered hardware.

Quoted 7 lines

How about we just move clk_enable/disable to runtime PM?

Since we use autosuspend delay, it theoretically has some impact, which is why I was refraining from doing so.

I can't decide if this impact would be marginal or significant.

Quoted 7 lines

I'd also refrain from doing this. Clock gating corresponds to the bigger part of the power savings from runtime power management, since it stops the dynamic power consumption and only leaves the static leakage. That said, the Hantro IP blocks have some internal clock gating as well, so it might not be as pronounced, depending on the custom vendor integration logic surrounding the Hantro hardware.

OK, I agree. We need to fix this issue then,
changing the order of the calls.

Quoted 8 lines

Actually even if autosuspend is not used, the runtime PM subsystem has some internal back-off mechanism based on measured power on and power off latencies. The driver should call pm_runtime_get_sync() first and then enable any necessary clocks. I can see that currently inside the resume callback we have some hardware accesses. If those really need to be there, they should be surrounded with appropriate clock enable and clock disable calls.

Currently, it's only used by imx8mq, and it's enclosed by clk_bulk_prepare_enable/disable_unprepare.

I am quite sure the prepare/unprepare usage is an oversight on our side, but it doesn't do any harm either.

Moving the clock handling to hantro_runtime_resume is possible, but looks like just low-hanging fruit.

Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 6 lines

I'm not seeing any code in CODA to handle this, so not sure how it's handling suspend/resume.

Maybe we can have CODA as the first user, given it's a well-maintained driver and should be fairly easy to test.

I remember checking a number of drivers using the m2m helpers randomly and none of them implemented suspend/resume correctly. I suppose that was not discovered because normally the userspace itself would stop the operation before the system is suspended, although it's not an API guarantee.

Quoted 7 lines
Quoted 2 lines

Indeed. Do you have any recomendations for how we could test this case to make sure we are handling it correctly?

I'd say that a simple offscreen command line gstreamer/ffmpeg decode with suspend/resume loop in another session should be able to trigger some issues.

I can try to fix the above clk/pm issue and take this helper on the same series, if that's useful.

Thanks,
Ezequiel

#11 lore raw

RFC PATCH v4 2/4 dt-bindings: mt8183: Added FD dt-bindings

In reply to Jerry-ch Chen

From: Jerry-ch Chen <[email protected]>

This patch adds DT binding documentation for the Face Detection (FD) unit of the Mediatek's mt8183 SoC.

---
 .../bindings/media/mediatek,mt8183-fd.txt     | 34 +++++++++++++++++++
 1 file changed, 34 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/mediatek,mt8183-fd.txt
Documentation/devicetree/bindings/media/mediatek,mt8183-fd.txt new file +34 -0
new file mode 100644
index 000000000000..46464175b95a
@@ -0,0 +1,34 @@
+* Mediatek Face Detection Unit (FD)
+
+Face Detection (FD) unit is a typical memory-to-memory HW device.
+It provides hardware accelerated face detection function, and it
+is able to detect different poses of faces. FD will writre result
+of detected face into memory as output.
+
+Required properties:
+- compatible: "mediatek,mt8183-fd"
+- mediatek,scp : the node of system control processor (SCP), see
+  Documentation/devicetree/bindings/remoteproc/mtk,scp.txt for details.
+- iommus: should point to the 3 entries:  M4U_PORT_CAM_FDVT_RP,
+  M4U_PORT_CAM_FDVT_WR and M4U_PORT_CAM_FDVT_RB.  (Please see
+  Documentation/devicetree/bindings/iommu/mediatek,iommu.txt for details.)
+- reg: Physical base address and length of the register space.
+- interrupts: interrupt number to the cpu.
+- clocks : must contain the FDVT clock, see
+  Documentation/devicetree/bindings/clock/clock-bindings.txt for details.
+- clock-names: must be "fd".
+- power-domain: must be "MT8183_POWER_DOMAIN_ISP".
+
+Example:
+	fd: fd@1502b000 {
+		compatible = "mediatek,mt8183-fd";
+		mediatek,scp = <&scp>;
+		iommus = <&iommu M4U_PORT_CAM_FDVT_RP>,
+			 <&iommu M4U_PORT_CAM_FDVT_WR>,
+			 <&iommu M4U_PORT_CAM_FDVT_RB>;
+		reg = <0 0x1502b000 0 0x1000>;
+		interrupts = <GIC_SPI 269 IRQ_TYPE_LEVEL_LOW>;
+		clocks = <&imgsys CLK_IMG_FDVT>;
+		clock-names = "fd";
+		power-domains = <&scpsys MT8183_POWER_DOMAIN_ISP>;
+	};
Signature · 1 line
2.18.0
maintainer
#12 lore raw

Re: [RFC PATCH V4 2/4] dt-bindings: mt8183: Added FD dt-bindings

Quoted 11 lines · Jerry-ch Chen

From: Jerry-ch Chen <[email protected]>

This patch adds DT binding documentation for the Face Detection (FD) unit of the Mediatek's mt8183 SoC.

---
 .../bindings/media/mediatek,mt8183-fd.txt     | 34 +++++++++++++++++++
 1 file changed, 34 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/mediatek,mt8183-fd.txt

Please add Acked-by/Reviewed-by tags when posting new versions. However, there's no need to repost patches *only* to add the tags. The upstream maintainer will do that for acks received on the version they apply.

If a tag was not added on purpose, please state why and what changed.

#13 lore raw

Re: [RFC PATCH V4 2/4] dt-bindings: mt8183: Added FD dt-bindings

Hi Rob,

Quoted 13 lines · Rob Herring
Quoted 11 lines · Jerry-ch Chen

From: Jerry-ch Chen <[email protected]>

This patch adds DT binding documentation for the Face Detection (FD) unit of the Mediatek's mt8183 SoC.

---
 .../bindings/media/mediatek,mt8183-fd.txt     | 34 +++++++++++++++++++
 1 file changed, 34 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/mediatek,mt8183-fd.txt
Quoted 5 lines

Please add Acked-by/Reviewed-by tags when posting new versions. However, there's no need to repost patches *only* to add the tags. The upstream maintainer will do that for acks received on the version they apply.

If a tag was not added on purpose, please state why and what changed.

Sorry for missing the tags,
I will add it back in next version.

Thanks and Best Regards,
Jerry

#14 lore raw

RFC PATCH v4 3/4 dts: arm64: mt8183: Add FD nodes

In reply to Jerry-ch Chen

From: Jerry-ch Chen <[email protected]>

This patch adds nodes for Face Detection (FD) unit. FD is embedded in Mediatek SoCs and works with the co-processor to perform face detection on the input data and image and output detected face result.

---
 arch/arm64/boot/dts/mediatek/mt8183.dtsi | 13 +++++++++++++
 1 file changed, 13 insertions(+)
arch/arm64/boot/dts/mediatek/mt8183.dtsi +13 -0
index c3a516e63141..6f31b5f4c17c 100644
@@ -432,6 +432,19 @@
 			#clock-cells = <1>;
 		};
 
+		fd:fd@1502b000 {
+			compatible = "mediatek,mt8183-fd";
+			mediatek,scp = <&scp>;
+			iommus = <&iommu M4U_PORT_CAM_FDVT_RP>,
+				 <&iommu M4U_PORT_CAM_FDVT_WR>,
+				 <&iommu M4U_PORT_CAM_FDVT_RB>;
+			reg = <0 0x1502b000 0 0x1000>;
+			interrupts = <GIC_SPI 269 IRQ_TYPE_LEVEL_LOW>;
+			clocks = <&imgsys CLK_IMG_FDVT>;
+			clock-names = "fd";
+			power-domains = <&scpsys MT8183_POWER_DOMAIN_ISP>;
+		};
+
 		vdecsys: syscon@16000000 {
 			compatible = "mediatek,mt8183-vdecsys", "syscon";
 			reg = <0 0x16000000 0 0x1000>;
Signature · 1 line
2.18.0
#15 lore raw

RFC PATCH v4 4/4 platform: mtk-isp: Add Mediatek FD driver

In reply to Jerry-ch Chen

From: Jerry-ch Chen <[email protected]>

This patch adds the driver of Face Detection (FD) unit in Mediatek camera system, providing face detection function.

The mtk-isp directory will contain drivers for multiple IP blocks found in Mediatek ISP system. It will include ISP Pass 1 driver (CAM), sensor interface driver, DIP driver and face detection driver.

---
 drivers/media/platform/Kconfig                |    2 +
 drivers/media/platform/Makefile               |    2 +
 drivers/media/platform/mtk-isp/fd/Kconfig     |   19 +
 drivers/media/platform/mtk-isp/fd/Makefile    |    5 +
 drivers/media/platform/mtk-isp/fd/mtk_fd.h    |  149 ++
 drivers/media/platform/mtk-isp/fd/mtk_fd_40.c | 1279 +++++++++++++++++
 include/uapi/linux/v4l2-controls.h            |    4 +
 include/uapi/linux/videodev2.h                |    3 +
 8 files changed, 1463 insertions(+)
 create mode 100644 drivers/media/platform/mtk-isp/fd/Kconfig
 create mode 100644 drivers/media/platform/mtk-isp/fd/Makefile
 create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd.h
 create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd_40.c
drivers/media/platform/Kconfig +2 -0
index a505e9f5a1e2..ae99258e26a5 100644
@@ -32,6 +32,8 @@ source "drivers/media/platform/davinci/Kconfig"
 
 source "drivers/media/platform/omap/Kconfig"
 
+source "drivers/media/platform/mtk-isp/fd/Kconfig"
+
 config VIDEO_ASPEED
 	tristate "Aspeed AST2400 and AST2500 Video Engine driver"
 	depends on VIDEO_V4L2
drivers/media/platform/Makefile +2 -0
index e6deb2597738..8b817cc80f43 100644
@@ -94,6 +94,8 @@ obj-$(CONFIG_VIDEO_MEDIATEK_MDP)	+= mtk-mdp/
 
 obj-$(CONFIG_VIDEO_MEDIATEK_JPEG)	+= mtk-jpeg/
 
+obj-$(CONFIG_VIDEO_MEDIATEK_FD)		+= mtk-isp/fd/
+
 obj-$(CONFIG_VIDEO_QCOM_CAMSS)		+= qcom/camss/
 
 obj-$(CONFIG_VIDEO_QCOM_VENUS)		+= qcom/venus/
drivers/media/platform/mtk-isp/fd/Kconfig new file +19 -0
new file mode 100644
index 000000000000..7a747d947e88
@@ -0,0 +1,19 @@
+config VIDEO_MEDIATEK_FD
+	tristate "Mediatek face detection processing function"
+	depends on VIDEO_V4L2
+	depends on ARCH_MEDIATEK
+	select VIDEOBUF2_DMA_CONTIG
+	select VIDEOBUF2_CORE
+	select VIDEOBUF2_V4L2
+	select VIDEOBUF2_MEMOPS
+	select MEDIA_CONTROLLER
+	select MTK_SCP
+
+	default n
+	help
+		Support the Face Detection (FD) feature in the Mediatek
+		mt8183 Soc.
+
+		FD driver is a V4L2 memory-to-memory device driver which
+		provides hardware accelerated face detection function,
+		it can detect different sizes of faces in a raw image.
drivers/media/platform/mtk-isp/fd/Makefile new file +5 -0
new file mode 100644
index 000000000000..9b1c501d73e1
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0
+
+mtk-fd-objs += mtk_fd_40.o
+
+obj-$(CONFIG_VIDEO_MEDIATEK_FD) += mtk-fd.o
\ No newline at end of file
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +149 -0

149 lines not on this page · raw message

drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +1279 -0

1,279 lines not on this page · raw message

include/uapi/linux/v4l2-controls.h +4 -0
index 3dcfc6148f99..eae876ea6d0a 100644
@@ -192,6 +192,10 @@ enum v4l2_colorfx {
  * We reserve 16 controls for this driver. */
 #define V4L2_CID_USER_IMX_BASE			(V4L2_CID_USER_BASE + 0x10b0)
 
+/* The base for the mediatek FD driver controls */
+/* We reserve 16 controls for this driver. */
+#define V4L2_CID_USER_MTK_FD_BASE		(V4L2_CID_USER_BASE + 0x10d0)
+
 /* MPEG-class control IDs */
 /* The MPEG controls are applicable to all codec controls
  * and the 'MPEG' part of the define is historical */
include/uapi/linux/videodev2.h +3 -0
index b5671ce2724f..203e49ebc9e8 100644
@@ -721,6 +721,9 @@ struct v4l2_pix_format {
 #define V4L2_META_FMT_UVC         v4l2_fourcc('U', 'V', 'C', 'H') /* UVC Payload Header metadata */
 #define V4L2_META_FMT_D4XX        v4l2_fourcc('D', '4', 'X', 'X') /* D4XX Payload Header metadata */
 
+/* Vendor specific - Mediatek Face Detection meta buffer format for firmware */
+#define V4L2_META_FMT_MTFD_RESULT  v4l2_fourcc('M', 'T', 'f', 'd') /* FD meta capture buffer */
+
 /* priv field value to indicates that subsequent fields are valid. */
 #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
 
Signature · 1 line
2.18.0
#16 lore raw

Re: [RFC PATCH V4 4/4] platform: mtk-isp: Add Mediatek FD driver

Hi Jerry,

Quoted 11 lines · Jerry-ch Chen

From: Jerry-ch Chen <[email protected]>

This patch adds the driver of Face Detection (FD) unit in Mediatek camera system, providing face detection function.

The mtk-isp directory will contain drivers for multiple IP blocks found in Mediatek ISP system. It will include ISP Pass 1 driver (CAM), sensor interface driver, DIP driver and face detection driver.

Quoted 15 lines
---
 drivers/media/platform/Kconfig                |    2 +
 drivers/media/platform/Makefile               |    2 +
 drivers/media/platform/mtk-isp/fd/Kconfig     |   19 +
 drivers/media/platform/mtk-isp/fd/Makefile    |    5 +
 drivers/media/platform/mtk-isp/fd/mtk_fd.h    |  149 ++
 drivers/media/platform/mtk-isp/fd/mtk_fd_40.c | 1279 +++++++++++++++++
 include/uapi/linux/v4l2-controls.h            |    4 +
 include/uapi/linux/videodev2.h                |    3 +
 8 files changed, 1463 insertions(+)
 create mode 100644 drivers/media/platform/mtk-isp/fd/Kconfig
 create mode 100644 drivers/media/platform/mtk-isp/fd/Makefile
 create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd.h
 create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd_40.c

Thank you for the patch. Please see my comments inline. Really sorry for the much delayed review.

[snip]

Quoted 5 lines
drivers/media/platform/mtk-isp/fd/Kconfig new file +5 -0
+config VIDEO_MEDIATEK_FD
+	tristate "Mediatek face detection processing function"
+	depends on VIDEO_V4L2
+	depends on ARCH_MEDIATEK
+	select VIDEOBUF2_DMA_CONTIG

drivers/media/platform/mtk-isp/fd/Kconfig line 15

Quoted 10 lines
drivers/media/platform/mtk-isp/fd/Kconfig new file +10 -0
+	select VIDEOBUF2_CORE
+	select VIDEOBUF2_V4L2
+	select VIDEOBUF2_MEMOPS
+	select MEDIA_CONTROLLER
+	select MTK_SCP
+
+	default n
+	help
+		Support the Face Detection (FD) feature in the Mediatek
+		mt8183 Soc.

MT8183?

drivers/media/platform/mtk-isp/fd/Kconfig line 19

Quoted 4 lines
drivers/media/platform/mtk-isp/fd/Kconfig new file +4 -0
+
+		FD driver is a V4L2 memory-to-memory device driver which
+		provides hardware accelerated face detection function,
+		it can detect different sizes of faces in a raw image.

A YUV image I guess?

[snip]

Quoted 23 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +17 -0
new file mode 100644
index 000000000000..d85bdcb70d6d
@@ -0,0 +1,149 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+//
+// Copyright (c) 2018 MediaTek Inc.
+
+#ifndef __MTK_FD_HW_H__
+#define __MTK_FD_HW_H__
+
+#include <linux/completion.h>
+#include <linux/io.h>
+#include <linux/types.h>
+#include <linux/platform_device.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/videobuf2-v4l2.h>
+
+#define MTK_FD_OUTPUT_MIN_WIDTH			26U
+#define MTK_FD_OUTPUT_MIN_HEIGHT		26U

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 27

Quoted 10 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +10 -0
+#define MTK_FD_OUTPUT_MAX_WIDTH			640U
+#define MTK_FD_OUTPUT_MAX_HEIGHT		480U
+
+#define MTK_FD_HW_FMT_VYUY			2
+#define MTK_FD_HW_FMT_UYVY			3
+#define MTK_FD_HW_FMT_YVYU			4
+#define MTK_FD_HW_FMT_YUYV			5
+#define MTK_FD_HW_FMT_YVU_2P			6
+#define MTK_FD_HW_FMT_YUV_2P			7
+#define MTK_FD_HW_FMT_UNKNOWN			8

What is an unknown format?

Quoted 8 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +8 -0
+
+#define MTK_FD_IPI_CMD_INIT			0
+#define MTK_FD_IPI_CMD_INIT_ACK			1
+#define MTK_FD_IPI_CMD_ENQUEUE			2
+#define MTK_FD_IPI_CMD_ENQ_ACK			3
+#define MTK_FD_IPI_CMD_EXIT			4
+#define MTK_FD_IPI_CMD_EXIT_ACK			5
+#define MTK_FD_IPI_CMD_RESET			6

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 45

Quoted 10 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +10 -0
+#define MTK_FD_IPI_CMD_RESET_ACK		7
+
+#define MTK_FD_REG_OFFSET_HW_ENABLE		0x4
+#define MTK_FD_REG_OFFSET_INT_EN		0x15c
+#define MTK_FD_REG_OFFSET_INT_VAL		0x168
+#define MTK_FD_REG_OFFSET_RESULT		0x178
+
+#define MTK_FD_SET_HW_ENABLE			0x111
+#define MTK_FD_RS_BUF_SIZE			2289664
+#define MTK_FD_HW_WORK_BUF_SIZE			0x100000

How about using the SZ_1M?

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 47

Quoted 2 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +2 -0
+#define MTK_FD_MAX_SPEEDUP			7
+#define MTK_FD_MAX_RESULT_NUM			1026

Perhaps NUM_RESULTS?

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 50

Quoted 3 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +3 -0
+
+/* Max scale size counts */
+#define MTK_FD_SCALE_ARR_NUM			15

Perhaps NUM_SCALE_SIZES?

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 52

Quoted 2 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +2 -0
+
+#define MTK_FD_HW_TIMEOUT			1000

What's the unit?

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 61

Quoted 9 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +9 -0
+
+enum face_angle {
+	MTK_FD_FACE_FRONT,
+	MTK_FD_FACE_RIGHT_50,
+	MTK_FD_FACE_LEFT_50,
+	MTK_FD_FACE_RIGHT_90,
+	MTK_FD_FACE_LEFT_90,
+	MTK_FD_FACE_ANGLE_NUM,
+};

This enum seems to define values for the V4L2_CID_MTK_FD_DETECT_POSE control. Considering that this is an enumeration and the values are actually integers (-90, -50, 0, 50, 90), perhaps this should be an INTEGER_MENU control instead?

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 74

Quoted 13 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +13 -0
+
+struct fd_buffer {
+	__u32 scp_addr;	/* used by SCP */
+	__u32 dma_addr;	/* used by DMA HW */
+} __packed;
+
+struct fd_face_result {
+	char data[16];
+};
+
+struct fd_user_output {
+	struct fd_face_result results[MTK_FD_MAX_RESULT_NUM];
+	__u16 number;

Is this perhaps the number of results? If so, would num_results be a better name?

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 75

Quoted 1 line
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +1 -0
+};

Since this struct is the meta buffer format, it is a part of the userspace interface and should be defined in a header under include/uapi/linux/.

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 84

Quoted 9 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +9 -0
+
+struct user_param {
+	u8 fd_speedup;
+	u8 fd_extra_model;
+	u8 scale_img_num;
+	u8 src_img_fmt;
+	__u16 scale_img_width[MTK_FD_SCALE_ARR_NUM];
+	__u16 scale_img_height[MTK_FD_SCALE_ARR_NUM];
+	__u16 face_directions[MTK_FD_FACE_ANGLE_NUM];

Is this a user-facing definition or an interface between the kernel driver and firmware? If the latter, the __ types shouldn't be used.

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 89

Quoted 5 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +5 -0
+} __packed;
+
+struct fd_init_param {
+	struct fd_buffer fd_manager;
+	__u32 rs_dma_addr;

Ditto.

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 93

Quoted 4 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +4 -0
+} __packed;
+
+struct fd_enq_param {
+	__u64 output_vaddr;

Ditto.

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 101

Quoted 8 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +8 -0
+	struct fd_buffer src_img[2];
+	struct fd_buffer user_result;
+	struct user_param user_param;
+} __packed;
+
+struct fd_ack_param {
+	__u32 ret_code;
+	__u32 ret_msg;

Ditto.

Quoted 1 line
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +1 -0
+} __packed;

[snip]

Quoted 1 line

+/* */

Was there supposed to be a comment here? :)

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 10 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +10 -0
+static int mtk_fd_hw_alloc_rs_dma_addr(struct mtk_fd_dev *fd)
+{
+	struct device *dev = fd->dev;
+	void *va;
+	dma_addr_t dma_handle;
+
+	va = dma_alloc_coherent(dev, MTK_FD_RS_BUF_SIZE, &dma_handle,
+				GFP_KERNEL);
+	if (!va) {
+		dev_err(dev, "dma_alloc null va\n");

No need to print errors for memory allocation failures, because one will be printed automatically.

[snip]

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 6 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +6 -0
+static int mtk_fd_hw_connect(struct mtk_fd_dev *fd)
+{
+	int ret;
+
+	ret = rproc_boot(fd->rproc_handle);
+

nit: Unnecessary blank line.

[snip]

Quoted 10 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +10 -0
+static int mtk_fd_vb2_queue_setup(struct vb2_queue *vq,
+				  unsigned int *num_buffers,
+				  unsigned int *num_planes,
+				  unsigned int sizes[],
+				  struct device *alloc_devs[])
+{
+	struct mtk_fd_ctx *ctx = vb2_get_drv_priv(vq);
+	unsigned int size[2];
+	unsigned int plane;
+

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 10 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +10 -0
+	switch (vq->type) {
+	case V4L2_BUF_TYPE_META_CAPTURE:
+		size[0] = ctx->dst_fmt.buffersize;
+		break;
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
+		size[0] = ctx->src_fmt.plane_fmt[0].sizeimage;
+		if (*num_planes == 2)
+			size[1] = ctx->src_fmt.plane_fmt[1].sizeimage;
+		break;
+	}

Is this code above needed? The code below sets sizes[] and it uses a for loop, without opencoded assignment for the second plane.

Quoted 22 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +22 -0
+
+	if (*num_planes > 2)
+		return -EINVAL;
+	if (*num_planes == 0) {
+		if (vq->type == V4L2_BUF_TYPE_META_CAPTURE) {
+			sizes[0] = ctx->dst_fmt.buffersize;
+			*num_planes = 1;
+			return 0;
+		}
+
+		*num_planes = ctx->src_fmt.num_planes;
+		for (plane = 0; plane < *num_planes; plane++)
+			sizes[plane] = ctx->src_fmt.plane_fmt[plane].sizeimage;
+		return 0;
+	}
+
+	for (plane = 0; plane < *num_planes; plane++) {
+		if (sizes[plane] < size[plane])
+			return -EINVAL;
+	}
+	return 0;
+}

[snip]

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 4 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +4 -0
+static int mtk_fd_enum_fmt_out_mp(struct file *file, void *fh,
+				  struct v4l2_fmtdesc *f)
+{
+	if (f->index >= NUM_FORMATS)

Please use ARRAY_SIZE().

Quoted 26 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +26 -0
+		return -EINVAL;
+
+	f->pixelformat = mtk_fd_img_fmts[f->index].pixelformat;
+	return 0;
+}
+
+static void mtk_fd_fill_pixfmt_mp(struct v4l2_pix_format_mplane *dfmt,
+				  const struct v4l2_pix_format_mplane *sfmt)
+{
+	dfmt->field = V4L2_FIELD_NONE;
+	dfmt->colorspace = V4L2_COLORSPACE_BT2020;
+	dfmt->num_planes = sfmt->num_planes;
+	dfmt->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
+	dfmt->quantization = V4L2_QUANTIZATION_DEFAULT;
+	dfmt->xfer_func =
+		V4L2_MAP_XFER_FUNC_DEFAULT(dfmt->colorspace);
+
+	/* Keep user setting as possible */
+	dfmt->width = clamp(dfmt->width,
+			    MTK_FD_OUTPUT_MIN_WIDTH,
+			    MTK_FD_OUTPUT_MAX_WIDTH);
+	dfmt->height = clamp(dfmt->height,
+			     MTK_FD_OUTPUT_MIN_HEIGHT,
+			     MTK_FD_OUTPUT_MAX_HEIGHT);
+
+	if (sfmt->num_planes == 2) {

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 10 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +10 -0
+		/* NV16M and NV61M has 1 byte per pixel */
+		dfmt->plane_fmt[0].bytesperline = dfmt->width;
+		dfmt->plane_fmt[1].bytesperline = dfmt->width;
+	} else {
+		/* 2 bytes per pixel */
+		dfmt->plane_fmt[0].bytesperline = dfmt->width * 2;
+	}
+
+	dfmt->plane_fmt[0].sizeimage =
+		dfmt->height * dfmt->plane_fmt[0].bytesperline;

Could some of the code above be replaced with v4l2_fill_pixfmt_mp()?

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 8 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +8 -0
+}
+
+static const struct v4l2_pix_format_mplane *mtk_fd_find_fmt(u32 format)
+{
+	unsigned int i;
+	const struct v4l2_pix_format_mplane *dev_fmt;
+
+	for (i = 0; i < NUM_FORMATS; i++) {

Please use ARRAY_SIZE rather than a custom macro.

Quoted 8 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +8 -0
+		dev_fmt = &mtk_fd_img_fmts[i];
+		if (dev_fmt->pixelformat == format)
+			return dev_fmt;
+	}
+
+	return NULL;
+}
+

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 10 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +10 -0
+static int mtk_fd_try_fmt_out_mp(struct file *file,
+				 void *fh,
+				 struct v4l2_format *f)
+{
+	struct v4l2_pix_format_mplane *pix_mp = &f->fmt.pix_mp;
+	const struct v4l2_pix_format_mplane *fmt;
+
+	fmt = mtk_fd_find_fmt(pix_mp->pixelformat);
+	if (!fmt)
+		fmt = &mtk_fd_img_fmts[0];	/* Get default img fmt */

nit: Please move the comment to a separate line and add braces.

Quoted 4 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +4 -0
+
+	mtk_fd_fill_pixfmt_mp(pix_mp, fmt);
+	return 0;
+}

[snip]

Quoted 7 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +7 -0
+static unsigned int get_fd_img_fmt(unsigned int fourcc)
+{
+	switch (fourcc) {
+	case V4L2_PIX_FMT_VYUY:
+		return MTK_FD_HW_FMT_VYUY;
+	case V4L2_PIX_FMT_YUYV:
+		return MTK_FD_HW_FMT_YUYV;

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 10 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +10 -0
+	case V4L2_PIX_FMT_YVYU:
+		return MTK_FD_HW_FMT_YVYU;
+	case V4L2_PIX_FMT_UYVY:
+		return MTK_FD_HW_FMT_UYVY;
+	case V4L2_PIX_FMT_NV16M:
+		return MTK_FD_HW_FMT_YUV_2P;
+	case V4L2_PIX_FMT_NV61M:
+		return MTK_FD_HW_FMT_YVU_2P;
+	default:
+		return MTK_FD_HW_FMT_UNKNOWN;

If we want to be paranoid, we can just add a WARN here and return 0. Shouldn't be a need to define an unknown format.

[snip]

Quoted 8 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +8 -0
+static void mtk_fd_fill_user_param(struct user_param *user_param,
+				   struct v4l2_ctrl_handler *hdl)
+{
+	struct v4l2_ctrl *ctrl;
+	int i;
+
+	ctrl = v4l2_ctrl_find(hdl, V4L2_CID_MTK_FD_SCALE_DOWN_IMG_WIDTH);
+	if (ctrl)

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 10 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +10 -0
+		for (i = 0; i < ctrl->elems; i++)
+			user_param->scale_img_width[i] = ctrl->p_new.p_u16[i];
+	ctrl = v4l2_ctrl_find(hdl, V4L2_CID_MTK_FD_SCALE_DOWN_IMG_HEIGHT);
+	if (ctrl)
+		for (i = 0; i < ctrl->elems; i++)
+			user_param->scale_img_height[i] = ctrl->p_new.p_u16[i];
+	ctrl = v4l2_ctrl_find(hdl, V4L2_CID_MTK_FD_SCALE_IMG_NUM);
+	if (ctrl)
+		user_param->scale_img_num = ctrl->val;
+

nit: Either separate the code dealing with all controls from each other, or none.

Quoted 19 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +19 -0
+	ctrl = v4l2_ctrl_find(hdl, V4L2_CID_MTK_FD_DETECT_POSE);
+	if (ctrl)
+		for (i = 0; i < ctrl->elems; i++)
+			user_param->face_directions[i] = ctrl->p_new.p_u16[i];
+	ctrl = v4l2_ctrl_find(hdl, V4L2_CID_MTK_FD_DETECT_SPEED);
+	if (ctrl)
+		user_param->fd_speedup = ctrl->val;
+	ctrl = v4l2_ctrl_find(hdl, V4L2_CID_MTK_FD_DETECTION_MODEL);
+	if (ctrl)
+		user_param->fd_extra_model = ctrl->val;
+}
+
+static void mtk_fd_device_run(void *priv)
+{
+	struct mtk_fd_ctx *ctx = priv;
+	struct mtk_fd_dev *fd = ctx->fd_dev;
+	struct vb2_v4l2_buffer *src_buf, *dst_buf;
+	struct fd_enq_param fd_param;
+	void *plane_vaddr;

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 10 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +10 -0
+
+	src_buf = v4l2_m2m_next_src_buf(ctx->fh.m2m_ctx);
+	dst_buf = v4l2_m2m_next_dst_buf(ctx->fh.m2m_ctx);
+
+	fd_param.src_img[0].dma_addr =
+		vb2_dma_contig_plane_dma_addr(&src_buf->vb2_buf, 0);
+	fd_param.user_result.dma_addr =
+		vb2_dma_contig_plane_dma_addr(&dst_buf->vb2_buf, 0);
+	plane_vaddr = vb2_plane_vaddr(&dst_buf->vb2_buf, 0);
+	fd_param.output_vaddr = (u64)(unsigned long)plane_vaddr;

Why is the vaddr needed here? Specfically, it could pose a security problem if CPU virtual addresses are exchanged with firmware.

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 5 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +5 -0
+	fd_param.user_param.src_img_fmt =
+		get_fd_img_fmt(ctx->src_fmt.pixelformat);
+	if (ctx->src_fmt.num_planes == 2)
+		fd_param.src_img[1].dma_addr =
+			vb2_dma_contig_plane_dma_addr(&src_buf->vb2_buf, 1);

nit: Could this be moved above, to be just below src_img[0] initialization, for readability reasons?

Quoted 94 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +94 -0
+	mtk_fd_fill_user_param(&fd_param.user_param, &ctx->hdl);
+
+	/* Complete request controls if any */
+	v4l2_ctrl_request_complete(src_buf->vb2_buf.req_obj.req, &ctx->hdl);
+
+	fd->output = plane_vaddr;
+	mtk_fd_hw_job_exec(fd, &fd_param);
+}
+
+static struct v4l2_m2m_ops fd_m2m_ops = {
+	.device_run = mtk_fd_device_run,
+};
+
+static const struct media_device_ops fd_m2m_media_ops = {
+	.req_validate	= vb2_request_validate,
+	.req_queue	= v4l2_m2m_request_queue,
+};
+
+static int mtk_fd_video_device_register(struct mtk_fd_dev *fd)
+{
+	struct video_device *vfd = &fd->vfd;
+	struct v4l2_m2m_dev *m2m_dev = fd->m2m_dev;
+	struct device *dev = fd->dev;
+	int ret;
+
+	vfd->fops = &fd_video_fops;
+	vfd->release = video_device_release;
+	vfd->lock = &fd->vfd_lock;
+	vfd->v4l2_dev = &fd->v4l2_dev;
+	vfd->vfl_dir = VFL_DIR_M2M;
+	vfd->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_VIDEO_OUTPUT_MPLANE |
+		V4L2_CAP_META_CAPTURE;
+	vfd->ioctl_ops = &mtk_fd_v4l2_video_out_ioctl_ops;
+
+	strscpy(vfd->name, dev_driver_string(dev), sizeof(vfd->name));
+
+	video_set_drvdata(vfd, fd);
+
+	ret = video_register_device(vfd, VFL_TYPE_GRABBER, 0);
+	if (ret) {
+		dev_err(dev, "Failed to register video device\n");
+		goto err_free_dev;
+	}
+
+	ret = v4l2_m2m_register_media_controller(m2m_dev, vfd,
+					     MEDIA_ENT_F_PROC_VIDEO_STATISTICS);
+	if (ret) {
+		dev_err(dev, "Failed to init mem2mem media controller\n");
+		goto err_unreg_video;
+	}
+	return 0;
+
+err_unreg_video:
+	video_unregister_device(vfd);
+err_free_dev:
+	video_device_release(vfd);
+	return ret;
+}
+
+static int mtk_fd_dev_v4l2_init(struct mtk_fd_dev *fd)
+{
+	struct media_device *mdev = &fd->mdev;
+	struct device *dev = fd->dev;
+	int ret;
+
+	ret = v4l2_device_register(dev, &fd->v4l2_dev);
+	if (ret) {
+		dev_err(dev, "Failed to register v4l2 device\n");
+		return ret;
+	}
+
+	fd->m2m_dev = v4l2_m2m_init(&fd_m2m_ops);
+	if (IS_ERR(fd->m2m_dev)) {
+		dev_err(dev, "Failed to init mem2mem device\n");
+		ret = PTR_ERR(fd->m2m_dev);
+		goto err_unreg_v4l2_dev;
+	}
+
+	mdev->dev = dev;
+	strscpy(mdev->model, dev_driver_string(dev), sizeof(mdev->model));
+	snprintf(mdev->bus_info, sizeof(mdev->bus_info),
+		 "platform:%s", dev_name(dev));
+	media_device_init(mdev);
+	mdev->ops = &fd_m2m_media_ops;
+	fd->v4l2_dev.mdev = mdev;
+
+	ret = mtk_fd_video_device_register(fd);
+	if (ret) {
+		dev_err(dev, "Failed to register video device\n");
+		goto err_cleanup_mdev;
+	}
+
+	ret = media_device_register(mdev);
+	if (ret) {

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 10 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +10 -0
+		dev_err(dev, "Failed to register mem2mem media device\n");
+		goto err_unreg_vdev;
+	}
+
+	return 0;
+
+err_unreg_vdev:
+	v4l2_m2m_unregister_media_controller(fd->m2m_dev);
+	video_unregister_device(&fd->vfd);
+	video_device_release(&fd->vfd);

This is inconsistent with the registration, which registers all of these in a function. Perhaps it would be worth to move this cleanup to a mtk_fd_video_device_unregister() function?

Quoted 41 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +41 -0
+err_cleanup_mdev:
+	media_device_cleanup(mdev);
+	v4l2_m2m_release(fd->m2m_dev);
+err_unreg_v4l2_dev:
+	v4l2_device_unregister(&fd->v4l2_dev);
+	return ret;
+}
+
+static void mtk_fd_dev_v4l2_release(struct mtk_fd_dev *fd)
+{
+	v4l2_m2m_unregister_media_controller(fd->m2m_dev);
+	video_unregister_device(&fd->vfd);
+	video_device_release(&fd->vfd);
+	media_device_cleanup(&fd->mdev);
+	v4l2_m2m_release(fd->m2m_dev);
+	v4l2_device_unregister(&fd->v4l2_dev);
+}
+
+static irqreturn_t mtk_fd_irq(int irq, void *data)
+{
+	struct mtk_fd_dev *fd = (struct mtk_fd_dev *)data;
+
+	/* must read this register otherwise HW will keep sending irq */
+	readl(fd->fd_base + MTK_FD_REG_OFFSET_INT_VAL);
+	fd->output->number = readl(fd->fd_base + MTK_FD_REG_OFFSET_RESULT);
+	dev_dbg(fd->dev, "mtk_fd_face_num:%d\n", fd->output->number);
+
+	mtk_fd_hw_done(fd, VB2_BUF_STATE_DONE);
+	return IRQ_HANDLED;
+}
+
+static int mtk_fd_hw_get_scp_mem(struct mtk_fd_dev *fd)
+{
+	struct device *dev = fd->dev;
+	dma_addr_t addr;
+	void *ptr;
+	u32 ret;
+
+	/*
+	 * Allocate coherent reserved memory for SCP firmware usage.
+	 * The size of SCP composer's memory is fixed to 0x100000

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 10 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +10 -0
+	 * for the requirement of firmware.
+	 */
+	ptr = dma_alloc_coherent(&fd->scp_pdev->dev,
+				 MTK_FD_HW_WORK_BUF_SIZE, &addr, GFP_KERNEL);
+	if (!ptr)
+		return -ENOMEM;
+
+	fd->scp_mem.scp_addr = addr;
+	fd->scp_mem_virt_addr = ptr;
+	dev_info(dev, "scp addr:%pad va:%pK\n", &addr, ptr);

These addresses are nothing a user should be concerned about, so please don't use the _info level here. If you think this is an important debugging information, please make it dev_dbg(). Otherwise, please just remove.

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 7 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +7 -0
+
+	/*
+	 * This reserved memory is also be used by FD HW.
+	 * Need to get iova address for FD DMA.
+	 */
+	addr = dma_map_resource(dev, addr, MTK_FD_HW_WORK_BUF_SIZE,
+				DMA_TO_DEVICE, DMA_ATTR_SKIP_CPU_SYNC);

Note that the second argument to dma_map_resource() is phys_addr_t, but the code above passes a DMA address. This works only by luck, because both physical and SCP DMA address space have the same addresses. To be fully correct, dma_map_single() needs to be used, with the kernel virtual address passed to it, but currently it doesn't handle the memory from a reserved pool. Thus, we can only keep the hack as is, but please add a comment explaining it, e.g.

/*
* FIXME: Assume SCP DMA and physical addresses are the same until
* dma_map_single() is fixed to handle reserved memory allocations. */

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 7 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +7 -0
+	if (dma_mapping_error(dev, addr)) {
+		dev_err(dev, "Failed to map scp iova\n");
+		ret = -ENOMEM;
+		goto fail_free_mem;
+	}
+	fd->scp_mem.dma_addr = addr;
+	dev_info(dev, "scp iova addr:%pad\n", &addr);

Ditto.

Best regards,
Tomasz

#17 lore raw

Re: [RFC PATCH V4 4/4] platform: mtk-isp: Add Mediatek FD driver

Hi Tomasz,

Quoted 30 lines · Tomasz Figa

Hi Jerry,

Quoted 11 lines · Jerry-ch Chen

From: Jerry-ch Chen <[email protected]>

This patch adds the driver of Face Detection (FD) unit in Mediatek camera system, providing face detection function.

The mtk-isp directory will contain drivers for multiple IP blocks found in Mediatek ISP system. It will include ISP Pass 1 driver (CAM), sensor interface driver, DIP driver and face detection driver.

Quoted 15 lines
---
 drivers/media/platform/Kconfig                |    2 +
 drivers/media/platform/Makefile               |    2 +
 drivers/media/platform/mtk-isp/fd/Kconfig     |   19 +
 drivers/media/platform/mtk-isp/fd/Makefile    |    5 +
 drivers/media/platform/mtk-isp/fd/mtk_fd.h    |  149 ++
 drivers/media/platform/mtk-isp/fd/mtk_fd_40.c | 1279 +++++++++++++++++
 include/uapi/linux/v4l2-controls.h            |    4 +
 include/uapi/linux/videodev2.h                |    3 +
 8 files changed, 1463 insertions(+)
 create mode 100644 drivers/media/platform/mtk-isp/fd/Kconfig
 create mode 100644 drivers/media/platform/mtk-isp/fd/Makefile
 create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd.h
 create mode 100644 drivers/media/platform/mtk-isp/fd/mtk_fd_40.c
Quoted 3 lines

Thank you for the patch. Please see my comments inline. Really sorry for the much delayed review.

I appreciate your comments, here's the reply

Quoted 10 lines

[snip]

Quoted 9 lines
drivers/media/platform/mtk-isp/fd/Kconfig new file +9 -0
+config VIDEO_MEDIATEK_FD
+	tristate "Mediatek face detection processing function"
+	depends on VIDEO_V4L2
+	depends on ARCH_MEDIATEK
+	select VIDEOBUF2_DMA_CONTIG
+	select VIDEOBUF2_CORE
+	select VIDEOBUF2_V4L2
+	select VIDEOBUF2_MEMOPS
+	select MEDIA_CONTROLLER
Quoted 8 lines

drivers/media/platform/mtk-isp/fd/Kconfig line 15

Quoted 6 lines
drivers/media/platform/mtk-isp/fd/Kconfig new file +6 -0
+	select MTK_SCP
+
+	default n
+	help
+		Support the Face Detection (FD) feature in the Mediatek
+		mt8183 Soc.

MT8183?

Fixed.

Quoted 7 lines

drivers/media/platform/mtk-isp/fd/Kconfig line 19

Quoted 5 lines
drivers/media/platform/mtk-isp/fd/Kconfig new file +4 -0
+
+		FD driver is a V4L2 memory-to-memory device driver which
+		provides hardware accelerated face detection function,
+		it can detect different sizes of faces in a raw image.

A YUV image I guess?

Fixed

Quoted 29 lines

[snip]

Quoted 27 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +21 -0
new file mode 100644
index 000000000000..d85bdcb70d6d
@@ -0,0 +1,149 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+//
+// Copyright (c) 2018 MediaTek Inc.
+
+#ifndef __MTK_FD_HW_H__
+#define __MTK_FD_HW_H__
+
+#include <linux/completion.h>
+#include <linux/io.h>
+#include <linux/types.h>
+#include <linux/platform_device.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/videobuf2-v4l2.h>
+
+#define MTK_FD_OUTPUT_MIN_WIDTH			26U
+#define MTK_FD_OUTPUT_MIN_HEIGHT		26U
+#define MTK_FD_OUTPUT_MAX_WIDTH			640U
+#define MTK_FD_OUTPUT_MAX_HEIGHT		480U
+
+#define MTK_FD_HW_FMT_VYUY			2
Quoted 8 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 27

Quoted 6 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +6 -0
+#define MTK_FD_HW_FMT_UYVY			3
+#define MTK_FD_HW_FMT_YVYU			4
+#define MTK_FD_HW_FMT_YUYV			5
+#define MTK_FD_HW_FMT_YVU_2P			6
+#define MTK_FD_HW_FMT_YUV_2P			7
+#define MTK_FD_HW_FMT_UNKNOWN			8

What is an unknown format?

It's used in translate v4l2 pixel format to the value that FD HW recognized. But I think the case of UNKNWOWN shouldn't happened. I will remove it

Quoted 13 lines
Quoted 13 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +12 -0
+
+#define MTK_FD_IPI_CMD_INIT			0
+#define MTK_FD_IPI_CMD_INIT_ACK			1
+#define MTK_FD_IPI_CMD_ENQUEUE			2
+#define MTK_FD_IPI_CMD_ENQ_ACK			3
+#define MTK_FD_IPI_CMD_EXIT			4
+#define MTK_FD_IPI_CMD_EXIT_ACK			5
+#define MTK_FD_IPI_CMD_RESET			6
+#define MTK_FD_IPI_CMD_RESET_ACK		7
+
+#define MTK_FD_REG_OFFSET_HW_ENABLE		0x4
+#define MTK_FD_REG_OFFSET_INT_EN		0x15c
Quoted 9 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 45

Quoted 6 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +6 -0
+#define MTK_FD_REG_OFFSET_INT_VAL		0x168
+#define MTK_FD_REG_OFFSET_RESULT		0x178
+
+#define MTK_FD_SET_HW_ENABLE			0x111
+#define MTK_FD_RS_BUF_SIZE			2289664
+#define MTK_FD_HW_WORK_BUF_SIZE			0x100000

How about using the SZ_1M?

yes, fixed.

Quoted 5 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 47

Quoted 2 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +2 -0
+#define MTK_FD_MAX_SPEEDUP			7
+#define MTK_FD_MAX_RESULT_NUM			1026

Perhaps NUM_RESULTS?

fixed as MTK_FD_MAX_NUM_RESULT.

Quoted 6 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 50

Quoted 3 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +3 -0
+
+/* Max scale size counts */
+#define MTK_FD_SCALE_ARR_NUM			15

Perhaps NUM_SCALE_SIZES?

fixed as MTK_FD_NUM_SCALE_SIZE

Quoted 4 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 52

Quoted 2 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +2 -0
+
+#define MTK_FD_HW_TIMEOUT			1000

What's the unit?

it's in milliseconds, I would like to rename it to MTK_FD_HW_TIMEOUT_MSEC

Quoted 16 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 61

Quoted 10 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +9 -0
+
+enum face_angle {
+	MTK_FD_FACE_FRONT,
+	MTK_FD_FACE_RIGHT_50,
+	MTK_FD_FACE_LEFT_50,
+	MTK_FD_FACE_RIGHT_90,
+	MTK_FD_FACE_LEFT_90,
+	MTK_FD_FACE_ANGLE_NUM,
+};

This enum seems to define values for the V4L2_CID_MTK_FD_DETECT_POSE control. Considering that this is an enumeration and the values are actually integers (-90, -50, 0, 50, 90), perhaps this should be an INTEGER_MENU control instead?

this ioctl let user select multiple face positions(combination of angles and directions) to be detected. so I thought I am not able to use the INTEGER_MENU for this purpose.

A bit-field as following should be used by user.
I consider adding it to uapi.

struct face_direction_def {
__u16 MTK_FD_FACE_DIR_0 : 1,

	MTK_FD_FACE_DIR_30 : 1,
	MTK_FD_FACE_DIR_60 : 1,
	MTK_FD_FACE_DIR_90 : 1,
	MTK_FD_FACE_DIR_120 : 1,
	MTK_FD_FACE_DIR_150 : 1,
	MTK_FD_FACE_DIR_180 : 1,
	MTK_FD_FACE_DIR_210 : 1,
	MTK_FD_FACE_DIR_240 : 1,
	MTK_FD_FACE_DIR_270 : 1,
	MTK_FD_FACE_DIR_300 : 1,
	MTK_FD_FACE_DIR_330 : 1,
	: 4;
};

User can also select some face directions of each face angle in one ioctl, for example:

/*
* u16 face_directions[MTK_FD_FACE_ANGLE_NUM] = {0}; *
* face_directions[MTK_FD_FACE_FRONT] = 0x7; //angle:0, dir:0,30,60
* face_directions[MTK_FACE_RIGHT_50] = 0x2; //angle:50, dir:30
*
*/

Quoted 5 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 66

Quoted 5 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +5 -0
+
+struct fd_buffer {
+	__u32 scp_addr;	/* used by SCP */
+	__u32 dma_addr;	/* used by DMA HW */
+} __packed;

fd buffer is used for scp ipi

Quoted 4 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 70

Quoted 4 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +4 -0
+
+struct fd_face_result {
+	char data[16];
+};

fd_face_result is used for user, so it should be moved to include/uapi/linux.
In fact, it has bit-field definition for user, so I would like to define it in include/uapi/linux as following:

struct fd_face_result {
__u64 face_idx : 12,

	type : 1,
	x0 : 10,
	y0 : 10,
	x1 : 10,
	y1 : 10,
	fcv1 : 11;
  __u64 fcv2 : 7,
	rip_dir : 4,
	rop_dir : 3,
	det_size : 5;
};
Quoted 8 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 74

Quoted 4 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +4 -0
+
+struct fd_user_output {
+	struct fd_face_result results[MTK_FD_MAX_RESULT_NUM];
+	__u16 number;

Is this perhaps the number of results? If so, would num_results be a better name?

yes, fixed.

Quoted 5 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 75

Quoted 1 line
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +1 -0
+};

Since this struct is the meta buffer format, it is a part of the userspace interface and should be defined in a header under include/uapi/linux/.

Ok, I will create include/uapi/linux/mtk_fd_40.h which suppose to include structures that userspace will use. should the private IOCTLs be placed in it together?

Quoted 4 lines
Quoted 4 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +4 -0
+
+struct user_param {
+	u8 fd_speedup;
+	u8 fd_extra_model;
Quoted 9 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 84

Quoted 5 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +5 -0
+	u8 scale_img_num;
+	u8 src_img_fmt;
+	__u16 scale_img_width[MTK_FD_SCALE_ARR_NUM];
+	__u16 scale_img_height[MTK_FD_SCALE_ARR_NUM];
+	__u16 face_directions[MTK_FD_FACE_ANGLE_NUM];

Is this a user-facing definition or an interface between the kernel driver and firmware? If the latter, the __ types shouldn't be used.

It's the later, I'll fix it.

Quoted 8 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 89

Quoted 5 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +5 -0
+} __packed;
+
+struct fd_init_param {
+	struct fd_buffer fd_manager;
+	__u32 rs_dma_addr;

Ditto.

done.

Quoted 6 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 93

Quoted 4 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +4 -0
+} __packed;
+
+struct fd_enq_param {
+	__u64 output_vaddr;

Ditto.

done.

Quoted 11 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 101

Quoted 9 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +8 -0
+	struct fd_buffer src_img[2];
+	struct fd_buffer user_result;
+	struct user_param user_param;
+} __packed;
+
+struct fd_ack_param {
+	__u32 ret_code;
+	__u32 ret_msg;

Ditto.

done.

Quoted 6 lines
Quoted 2 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +1 -0
+} __packed;

[snip]

Quoted 1 line

+/* */

Was there supposed to be a comment here? :)

Ooops, no, it should be removed.

Quoted 14 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 11 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +10 -0
+static int mtk_fd_hw_alloc_rs_dma_addr(struct mtk_fd_dev *fd)
+{
+	struct device *dev = fd->dev;
+	void *va;
+	dma_addr_t dma_handle;
+
+	va = dma_alloc_coherent(dev, MTK_FD_RS_BUF_SIZE, &dma_handle,
+				GFP_KERNEL);
+	if (!va) {
+		dev_err(dev, "dma_alloc null va\n");

No need to print errors for memory allocation failures, because one will be printed automatically.

Ok, log removed.

Quoted 11 lines

[snip]

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 6 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +6 -0
+static int mtk_fd_hw_connect(struct mtk_fd_dev *fd)
+{
+	int ret;
+
+	ret = rproc_boot(fd->rproc_handle);
+

nit: Unnecessary blank line.

fixed.

Quoted 16 lines

[snip]

Quoted 15 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +15 -0
+static int mtk_fd_vb2_queue_setup(struct vb2_queue *vq,
+				  unsigned int *num_buffers,
+				  unsigned int *num_planes,
+				  unsigned int sizes[],
+				  struct device *alloc_devs[])
+{
+	struct mtk_fd_ctx *ctx = vb2_get_drv_priv(vq);
+	unsigned int size[2];
+	unsigned int plane;
+
+	switch (vq->type) {
+	case V4L2_BUF_TYPE_META_CAPTURE:
+		size[0] = ctx->dst_fmt.buffersize;
+		break;
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
Quoted 9 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 5 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +5 -0
+		size[0] = ctx->src_fmt.plane_fmt[0].sizeimage;
+		if (*num_planes == 2)
+			size[1] = ctx->src_fmt.plane_fmt[1].sizeimage;
+		break;
+	}

Is this code above needed? The code below sets sizes[] and it uses a for loop, without opencoded assignment for the second plane.

Looks like not really useful here,
it should check sizes and num_planes if num_plane not zero, and for V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE, it will at most have 2 planes, maybe no need for loop as well.
I will refine this function as following:
mtk_fd_vb2_queue_setup(...)

{
	struct mtk_fd_ctx *ctx = vb2_get_drv_priv(vq);

	if (*num_planes == 0) {
		if (vq->type == V4L2_BUF_TYPE_META_CAPTURE) {
			sizes[0] = ctx->dst_fmt.buffersize;
			*num_planes = 1;
			return 0;
		} else if (vq->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) {
			*num_planes = ctx->src_fmt.num_planes;
			sizes[0] = ctx->src_fmt.plane_fmt[0].sizeimage;
			if (*num_planes == 2)
				sizes[1] = ctx->src_fmt.plane_fmt[1].sizeimage;
			return 0;
		}
		return -EINVAL;
	}

	/* If num_plane not zero, check the num_plane and sizes*/
	if (vq->type == V4L2_BUF_TYPE_META_CAPTURE) {
		if ((*num_planes == 1) &&
		    (sizes[0] <= ctx->dst_fmt.buffersize))
			return 0;
		else
			return -EINVAL;
	}
	if (vq->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) {
		if ((*num_planes == 1) &&
		    (sizes[0] <= ctx->src_fmt.plane_fmt[0].sizeimage))
			return 0;
		else if ((*num_planes == 2) &&
			 (sizes[0] <= ctx->src_fmt.plane_fmt[0].sizeimage) &&
			 (sizes[1] <= ctx->src_fmt.plane_fmt[1].sizeimage))
			return 0;
		else
			return -EINVAL;

	}
	return 0;
}
Quoted 23 lines
Quoted 22 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +22 -0
+
+	if (*num_planes > 2)
+		return -EINVAL;
+	if (*num_planes == 0) {
+		if (vq->type == V4L2_BUF_TYPE_META_CAPTURE) {
+			sizes[0] = ctx->dst_fmt.buffersize;
+			*num_planes = 1;
+			return 0;
+		 }
+
+		*num_planes = ctx->src_fmt.num_planes;
+		for (plane = 0; plane < *num_planes; plane++)
+			sizes[plane] = ctx->src_fmt.plane_fmt[plane].sizeimage;
+		return 0;
+	}
+
+	for (plane = 0; plane < *num_planes; plane++) {
+		if (sizes[plane] < size[plane])
+			return -EINVAL;
+	}
+	return 0;
+}

[snip]

Quoted 6 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 4 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +4 -0
+static int mtk_fd_enum_fmt_out_mp(struct file *file, void *fh,
+				  struct v4l2_fmtdesc *f)
+{
+	if (f->index >= NUM_FORMATS)

Please use ARRAY_SIZE().

fixed.

Quoted 31 lines
Quoted 31 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +30 -0
+		return -EINVAL;
+
+	f->pixelformat = mtk_fd_img_fmts[f->index].pixelformat;
+	return 0;
+}
+
+static void mtk_fd_fill_pixfmt_mp(struct v4l2_pix_format_mplane *dfmt,
+				  const struct v4l2_pix_format_mplane *sfmt)
+{
+	dfmt->field = V4L2_FIELD_NONE;
+	dfmt->colorspace = V4L2_COLORSPACE_BT2020;
+	dfmt->num_planes = sfmt->num_planes;
+	dfmt->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
+	dfmt->quantization = V4L2_QUANTIZATION_DEFAULT;
+	dfmt->xfer_func =
+		V4L2_MAP_XFER_FUNC_DEFAULT(dfmt->colorspace);
+
+	/* Keep user setting as possible */
+	dfmt->width = clamp(dfmt->width,
+			    MTK_FD_OUTPUT_MIN_WIDTH,
+			    MTK_FD_OUTPUT_MAX_WIDTH);
+	dfmt->height = clamp(dfmt->height,
+			     MTK_FD_OUTPUT_MIN_HEIGHT,
+			     MTK_FD_OUTPUT_MAX_HEIGHT);
+
+	if (sfmt->num_planes == 2) {
+		/* NV16M and NV61M has 1 byte per pixel */
+		dfmt->plane_fmt[0].bytesperline = dfmt->width;
+		dfmt->plane_fmt[1].bytesperline = dfmt->width;
+	} else {
Quoted 9 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 6 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +6 -0
+		/* 2 bytes per pixel */
+		dfmt->plane_fmt[0].bytesperline = dfmt->width * 2;
+	}
+
+	dfmt->plane_fmt[0].sizeimage =
+		dfmt->height * dfmt->plane_fmt[0].bytesperline;

Could some of the code above be replaced with v4l2_fill_pixfmt_mp()?

I would like to refine as following

mtk_fd_fill_pixfmt_mp(...){
	v4l2_fill_pixfmt_mp(dfmt, sfmt->pixelformat, dfmt->width,

dfmt->height);

	dfmt->field = V4L2_FIELD_NONE;
	dfmt->colorspace = V4L2_COLORSPACE_BT2020;
	dfmt->num_planes = sfmt->num_planes;
	dfmt->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
	dfmt->quantization = V4L2_QUANTIZATION_DEFAULT;
	dfmt->xfer_func =
		V4L2_MAP_XFER_FUNC_DEFAULT(dfmt->colorspace);
}
Quoted 10 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 8 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +8 -0
+}
+
+static const struct v4l2_pix_format_mplane *mtk_fd_find_fmt(u32 format)
+{
+	unsigned int i;
+	const struct v4l2_pix_format_mplane *dev_fmt;
+
+	for (i = 0; i < NUM_FORMATS; i++) {

Please use ARRAY_SIZE rather than a custom macro.

fixed.

Quoted 13 lines
Quoted 13 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +12 -0
+		dev_fmt = &mtk_fd_img_fmts[i];
+		if (dev_fmt->pixelformat == format)
+			return dev_fmt;
+	}
+
+	return NULL;
+}
+
+static int mtk_fd_try_fmt_out_mp(struct file *file,
+				 void *fh,
+				 struct v4l2_format *f)
+{
Quoted 9 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 6 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +6 -0
+	struct v4l2_pix_format_mplane *pix_mp = &f->fmt.pix_mp;
+	const struct v4l2_pix_format_mplane *fmt;
+
+	fmt = mtk_fd_find_fmt(pix_mp->pixelformat);
+	if (!fmt)
+		fmt = &mtk_fd_img_fmts[0];	/* Get default img fmt */

nit: Please move the comment to a separate line and add braces.

fixed.

Quoted 17 lines
Quoted 4 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +4 -0
+
+	mtk_fd_fill_pixfmt_mp(pix_mp, fmt);
+	return 0;
+}

[snip]

Quoted 12 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +12 -0
+static unsigned int get_fd_img_fmt(unsigned int fourcc)
+{
+	switch (fourcc) {
+	case V4L2_PIX_FMT_VYUY:
+		return MTK_FD_HW_FMT_VYUY;
+	case V4L2_PIX_FMT_YUYV:
+		return MTK_FD_HW_FMT_YUYV;
+	case V4L2_PIX_FMT_YVYU:
+		return MTK_FD_HW_FMT_YVYU;
+	case V4L2_PIX_FMT_UYVY:
+		return MTK_FD_HW_FMT_UYVY;
+	case V4L2_PIX_FMT_NV16M:
Quoted 8 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 5 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +5 -0
+		return MTK_FD_HW_FMT_YUV_2P;
+	case V4L2_PIX_FMT_NV61M:
+		return MTK_FD_HW_FMT_YVU_2P;
+	default:
+		return MTK_FD_HW_FMT_UNKNOWN;

If we want to be paranoid, we can just add a WARN here and return 0. Shouldn't be a need to define an unknown format.

Ok, I will refine as following:

default:

	dev_warn(dev, "%s: Unsupported V4L2_PIX_FMT\n", __func__);
	return 0;
Quoted 15 lines

[snip]

Quoted 13 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +13 -0
+static void mtk_fd_fill_user_param(struct user_param *user_param,
+				   struct v4l2_ctrl_handler *hdl)
+{
+	struct v4l2_ctrl *ctrl;
+	int i;
+
+	ctrl = v4l2_ctrl_find(hdl, V4L2_CID_MTK_FD_SCALE_DOWN_IMG_WIDTH);
+	if (ctrl)
+		for (i = 0; i < ctrl->elems; i++)
+			user_param->scale_img_width[i] = ctrl->p_new.p_u16[i];
+	ctrl = v4l2_ctrl_find(hdl, V4L2_CID_MTK_FD_SCALE_DOWN_IMG_HEIGHT);
+	if (ctrl)
+		for (i = 0; i < ctrl->elems; i++)
Quoted 9 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 5 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +5 -0
+			user_param->scale_img_height[i] = ctrl->p_new.p_u16[i];
+	ctrl = v4l2_ctrl_find(hdl, V4L2_CID_MTK_FD_SCALE_IMG_NUM);
+	if (ctrl)
+		user_param->scale_img_num = ctrl->val;
+

nit: Either separate the code dealing with all controls from each other, or none.

Fixed.

Quoted 24 lines
Quoted 24 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +24 -0
+	ctrl = v4l2_ctrl_find(hdl, V4L2_CID_MTK_FD_DETECT_POSE);
+	if (ctrl)
+		for (i = 0; i < ctrl->elems; i++)
+			user_param->face_directions[i] = ctrl->p_new.p_u16[i];
+	ctrl = v4l2_ctrl_find(hdl, V4L2_CID_MTK_FD_DETECT_SPEED);
+	if (ctrl)
+		user_param->fd_speedup = ctrl->val;
+	ctrl = v4l2_ctrl_find(hdl, V4L2_CID_MTK_FD_DETECTION_MODEL);
+	if (ctrl)
+		user_param->fd_extra_model = ctrl->val;
+}
+
+static void mtk_fd_device_run(void *priv)
+{
+	struct mtk_fd_ctx *ctx = priv;
+	struct mtk_fd_dev *fd = ctx->fd_dev;
+	struct vb2_v4l2_buffer *src_buf, *dst_buf;
+	struct fd_enq_param fd_param;
+	void *plane_vaddr;
+
+	src_buf = v4l2_m2m_next_src_buf(ctx->fh.m2m_ctx);
+	dst_buf = v4l2_m2m_next_dst_buf(ctx->fh.m2m_ctx);
+
+	fd_param.src_img[0].dma_addr =
Quoted 9 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 5 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +5 -0
+		vb2_dma_contig_plane_dma_addr(&src_buf->vb2_buf, 0);
+	fd_param.user_result.dma_addr =
+		vb2_dma_contig_plane_dma_addr(&dst_buf->vb2_buf, 0);
+	plane_vaddr = vb2_plane_vaddr(&dst_buf->vb2_buf, 0);
+	fd_param.output_vaddr = (u64)(unsigned long)plane_vaddr;

Why is the vaddr needed here? Specfically, it could pose a security problem if CPU virtual addresses are exchanged with firmware.

This should be removed... SCP side as well.

Quoted 9 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 5 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +5 -0
+	fd_param.user_param.src_img_fmt =
+		get_fd_img_fmt(ctx->src_fmt.pixelformat);
+	if (ctx->src_fmt.num_planes == 2)
+		fd_param.src_img[1].dma_addr =
+			vb2_dma_contig_plane_dma_addr(&src_buf->vb2_buf, 1);

nit: Could this be moved above, to be just below src_img[0] initialization, for readability reasons?

Ok, this function will be refined as

static void mtk_fd_device_run(void *priv)

{
	struct mtk_fd_ctx *ctx = priv;
	struct mtk_fd_dev *fd = ctx->fd_dev;
	struct vb2_v4l2_buffer *src_buf, *dst_buf;
	struct fd_enq_param fd_param;

	src_buf = v4l2_m2m_next_src_buf(ctx->fh.m2m_ctx);
	dst_buf = v4l2_m2m_next_dst_buf(ctx->fh.m2m_ctx);

	fd_param.src_img[0].dma_addr =
		vb2_dma_contig_plane_dma_addr(&src_buf->vb2_buf, 0);
	if (ctx->src_fmt.num_planes == 2)
		fd_param.src_img[1].dma_addr =
			vb2_dma_contig_plane_dma_addr(&src_buf->vb2_buf, 1);
	fd_param.user_result.dma_addr =
		vb2_dma_contig_plane_dma_addr(&dst_buf->vb2_buf, 0);
	fd_param.user_param.src_img_fmt =
		get_fd_img_fmt(fd->dev, ctx->src_fmt.pixelformat);

	mtk_fd_fill_user_param(&fd_param.user_param, &ctx->hdl);

	/* Complete request controls if any */
	v4l2_ctrl_request_complete(src_buf->vb2_buf.req_obj.req, &ctx->hdl);

	fd->output = vb2_plane_vaddr(&dst_buf->vb2_buf, 0);
	mtk_fd_hw_job_exec(fd, &fd_param);
}
Quoted 100 lines
Quoted 100 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +100 -0
+	mtk_fd_fill_user_param(&fd_param.user_param, &ctx->hdl);
+
+	/* Complete request controls if any */
+	v4l2_ctrl_request_complete(src_buf->vb2_buf.req_obj.req, &ctx->hdl);
+
+	fd->output = plane_vaddr;
+	mtk_fd_hw_job_exec(fd, &fd_param);
+}
+
+static struct v4l2_m2m_ops fd_m2m_ops = {
+	.device_run = mtk_fd_device_run,
+};
+
+static const struct media_device_ops fd_m2m_media_ops = {
+	.req_validate	= vb2_request_validate,
+	.req_queue	= v4l2_m2m_request_queue,
+};
+
+static int mtk_fd_video_device_register(struct mtk_fd_dev *fd)
+{
+	struct video_device *vfd = &fd->vfd;
+	struct v4l2_m2m_dev *m2m_dev = fd->m2m_dev;
+	struct device *dev = fd->dev;
+	int ret;
+
+	vfd->fops = &fd_video_fops;
+	vfd->release = video_device_release;
+	vfd->lock = &fd->vfd_lock;
+	vfd->v4l2_dev = &fd->v4l2_dev;
+	vfd->vfl_dir = VFL_DIR_M2M;
+	vfd->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_VIDEO_OUTPUT_MPLANE |
+		V4L2_CAP_META_CAPTURE;
+	vfd->ioctl_ops = &mtk_fd_v4l2_video_out_ioctl_ops;
+
+	strscpy(vfd->name, dev_driver_string(dev), sizeof(vfd->name));
+
+	video_set_drvdata(vfd, fd);
+
+	ret = video_register_device(vfd, VFL_TYPE_GRABBER, 0);
+	if (ret) {
+		dev_err(dev, "Failed to register video device\n");
+		goto err_free_dev;
+	}
+
+	ret = v4l2_m2m_register_media_controller(m2m_dev, vfd,
+					     MEDIA_ENT_F_PROC_VIDEO_STATISTICS);
+	if (ret) {
+		dev_err(dev, "Failed to init mem2mem media controller\n");
+		goto err_unreg_video;
+	}
+	return 0;
+
+err_unreg_video:
+	video_unregister_device(vfd);
+err_free_dev:
+	video_device_release(vfd);
+	return ret;
+}
+
+static int mtk_fd_dev_v4l2_init(struct mtk_fd_dev *fd)
+{
+	struct media_device *mdev = &fd->mdev;
+	struct device *dev = fd->dev;
+	int ret;
+
+	ret = v4l2_device_register(dev, &fd->v4l2_dev);
+	if (ret) {
+		dev_err(dev, "Failed to register v4l2 device\n");
+		return ret;
+	}
+
+	fd->m2m_dev = v4l2_m2m_init(&fd_m2m_ops);
+	if (IS_ERR(fd->m2m_dev)) {
+		dev_err(dev, "Failed to init mem2mem device\n");
+		ret = PTR_ERR(fd->m2m_dev);
+		goto err_unreg_v4l2_dev;
+	}
+
+	mdev->dev = dev;
+	strscpy(mdev->model, dev_driver_string(dev), sizeof(mdev->model));
+	snprintf(mdev->bus_info, sizeof(mdev->bus_info),
+		 "platform:%s", dev_name(dev));
+	media_device_init(mdev);
+	mdev->ops = &fd_m2m_media_ops;
+	fd->v4l2_dev.mdev = mdev;
+
+	ret = mtk_fd_video_device_register(fd);
+	if (ret) {
+		dev_err(dev, "Failed to register video device\n");
+		goto err_cleanup_mdev;
+	}
+
+	ret = media_device_register(mdev);
+	if (ret) {
+		dev_err(dev, "Failed to register mem2mem media device\n");
+		goto err_unreg_vdev;
+	}
+
+	return 0;
+
Quoted 9 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 4 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +4 -0
+err_unreg_vdev:
+	v4l2_m2m_unregister_media_controller(fd->m2m_dev);
+	video_unregister_device(&fd->vfd);
+	video_device_release(&fd->vfd);

This is inconsistent with the registration, which registers all of these in a function. Perhaps it would be worth to move this cleanup to a mtk_fd_video_device_unregister() function?

Fixed.

Quoted 47 lines
Quoted 47 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +47 -0
+err_cleanup_mdev:
+	media_device_cleanup(mdev);
+	v4l2_m2m_release(fd->m2m_dev);
+err_unreg_v4l2_dev:
+	v4l2_device_unregister(&fd->v4l2_dev);
+	return ret;
+}
+
+static void mtk_fd_dev_v4l2_release(struct mtk_fd_dev *fd)
+{
+	v4l2_m2m_unregister_media_controller(fd->m2m_dev);
+	video_unregister_device(&fd->vfd);
+	video_device_release(&fd->vfd);
+	media_device_cleanup(&fd->mdev);
+	v4l2_m2m_release(fd->m2m_dev);
+	v4l2_device_unregister(&fd->v4l2_dev);
+}
+
+static irqreturn_t mtk_fd_irq(int irq, void *data)
+{
+	struct mtk_fd_dev *fd = (struct mtk_fd_dev *)data;
+
+	/* must read this register otherwise HW will keep sending irq */
+	readl(fd->fd_base + MTK_FD_REG_OFFSET_INT_VAL);
+	fd->output->number = readl(fd->fd_base + MTK_FD_REG_OFFSET_RESULT);
+	dev_dbg(fd->dev, "mtk_fd_face_num:%d\n", fd->output->number);
+
+	mtk_fd_hw_done(fd, VB2_BUF_STATE_DONE);
+	return IRQ_HANDLED;
+}
+
+static int mtk_fd_hw_get_scp_mem(struct mtk_fd_dev *fd)
+{
+	struct device *dev = fd->dev;
+	dma_addr_t addr;
+	void *ptr;
+	u32 ret;
+
+	/*
+	 * Allocate coherent reserved memory for SCP firmware usage.
+	 * The size of SCP composer's memory is fixed to 0x100000
+	 * for the requirement of firmware.
+	 */
+	ptr = dma_alloc_coherent(&fd->scp_pdev->dev,
+				 MTK_FD_HW_WORK_BUF_SIZE, &addr, GFP_KERNEL);
+	if (!ptr)
+		return -ENOMEM;
Quoted 9 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 4 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +4 -0
+
+	fd->scp_mem.scp_addr = addr;
+	fd->scp_mem_virt_addr = ptr;
+	dev_info(dev, "scp addr:%pad va:%pK\n", &addr, ptr);

These addresses are nothing a user should be concerned about, so please don't use the _info level here. If you think this is an important debugging information, please make it dev_dbg(). Otherwise, please just remove.

Removed.

Quoted 16 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 7 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +7 -0
+
+	/*
+	 * This reserved memory is also be used by FD HW.
+	 * Need to get iova address for FD DMA.
+	 */
+	addr = dma_map_resource(dev, addr, MTK_FD_HW_WORK_BUF_SIZE,
+				DMA_TO_DEVICE, DMA_ATTR_SKIP_CPU_SYNC);

Note that the second argument to dma_map_resource() is phys_addr_t, but the code above passes a DMA address. This works only by luck, because both physical and SCP DMA address space have the same addresses. To be fully correct, dma_map_single() needs to be used, with the kernel virtual address passed to it, but currently it doesn't handle the memory from a reserved pool. Thus, we can only keep the hack as is, but please add a comment explaining it, e.g.

Quoted 5 lines

/*
* FIXME: Assume SCP DMA and physical addresses are the same until
* dma_map_single() is fixed to handle reserved memory allocations. */

Ok, I've insert it above the comments there.
as following:

/*
* FIXME: Assume SCP DMA and physical addresses are the same until
* dma_map_single() is fixed to handle reserved memory allocations. *
* This reserved memory will also be used by FD HW.
* Need to get iova address for FD DMA.
*/

Quoted 9 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h

Quoted 7 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +7 -0
+	if (dma_mapping_error(dev, addr)) {
+		dev_err(dev, "Failed to map scp iova\n");
+		ret = -ENOMEM;
+		goto fail_free_mem;
+	}
+	fd->scp_mem.dma_addr = addr;
+	dev_info(dev, "scp iova addr:%pad\n", &addr);

Ditto.

fixed.

Quoted 3 lines

Best regards,
Tomasz

Thanks and best regards,
Jerry

#18 lore raw

Re: [RFC PATCH V4 4/4] platform: mtk-isp: Add Mediatek FD driver

r

Quoted 7 lines · Jerry-ch Chen

Hi Tomasz,

Quoted 3 lines · Tomasz Figa

Hi Jerry,

On Wed, Dec 04, 2019 at 08:47:32PM +0800, Jerry-ch Chen wrote:

[snip]

Quoted 16 lines
Quoted 6 lines
Quoted 6 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +6 -0
+
+enum face_angle {
+   MTK_FD_FACE_FRONT,
+   MTK_FD_FACE_RIGHT_50,
+   MTK_FD_FACE_LEFT_50,
+   MTK_FD_FACE_RIGHT_90,
Quoted 9 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 61

Quoted 3 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +3 -0
+   MTK_FD_FACE_LEFT_90,
+   MTK_FD_FACE_ANGLE_NUM,
+};

This enum seems to define values for the V4L2_CID_MTK_FD_DETECT_POSE control. Considering that this is an enumeration and the values are actually integers (-90, -50, 0, 50, 90), perhaps this should be an INTEGER_MENU control instead?

Quoted 3 lines

this ioctl let user select multiple face positions(combination of angles and directions) to be detected. so I thought I am not able to use the INTEGER_MENU for this purpose.

Ah, okay, I thought there is only one selection possible.

Quoted 6 lines

A bit-field as following should be used by user.
I consider adding it to uapi.

struct face_direction_def {
__u16 MTK_FD_FACE_DIR_0 : 1,

Quoted 13 lines
        MTK_FD_FACE_DIR_30 : 1,
        MTK_FD_FACE_DIR_60 : 1,
        MTK_FD_FACE_DIR_90 : 1,
        MTK_FD_FACE_DIR_120 : 1,
        MTK_FD_FACE_DIR_150 : 1,
        MTK_FD_FACE_DIR_180 : 1,
        MTK_FD_FACE_DIR_210 : 1,
        MTK_FD_FACE_DIR_240 : 1,
        MTK_FD_FACE_DIR_270 : 1,
        MTK_FD_FACE_DIR_300 : 1,
        MTK_FD_FACE_DIR_330 : 1,
        : 4;
};

Note that bit fields are not recommended in UAPI because of not being well specified by the language. Instead bits should be defined using macros with explicit masks or sometimes enums.

Quoted 11 lines

User can also select some face directions of each face angle in one ioctl, for example:

/*
* u16 face_directions[MTK_FD_FACE_ANGLE_NUM] = {0}; *
* face_directions[MTK_FD_FACE_FRONT] = 0x7; //angle:0, dir:0,30,60
* face_directions[MTK_FACE_RIGHT_50] = 0x2; //angle:50, dir:30
*
*/

Makes sense, thanks.

Quoted 19 lines
Quoted 6 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 66

Quoted 6 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +5 -0
+
+struct fd_buffer {
+   __u32 scp_addr; /* used by SCP */
+   __u32 dma_addr; /* used by DMA HW */
+} __packed;

fd buffer is used for scp ipi

Quoted 4 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 70

Quoted 4 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +4 -0
+
+struct fd_face_result {
+   char data[16];
+};

fd_face_result is used for user, so it should be moved to include/uapi/linux.
In fact, it has bit-field definition for user, so I would like to define it in include/uapi/linux as following:

struct fd_face_result {
__u64 face_idx : 12,

Quoted 12 lines
        type : 1,
        x0 : 10,
        y0 : 10,
        x1 : 10,
        y1 : 10,
        fcv1 : 11;
  __u64 fcv2 : 7,
        rip_dir : 4,
        rop_dir : 3,
        det_size : 5;
};

Indeed this should be defined, but as per my comment above, please avoid using the bitfield construct and define shifts and masks instead.

Quoted 10 lines
Quoted 9 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 74

Quoted 5 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +4 -0
+
+struct fd_user_output {
+   struct fd_face_result results[MTK_FD_MAX_RESULT_NUM];
+   __u16 number;

Is this perhaps the number of results? If so, would num_results be a better name?

yes, fixed.

Quoted 9 lines
Quoted 5 lines
Quoted 1 line

+};

Since this struct is the meta buffer format, it is a part of the userspace interface and should be defined in a header under include/uapi/linux/.

Ok, I will create include/uapi/linux/mtk_fd_40.h which suppose to include structures that userspace will use. should the private IOCTLs be placed in it together?

Sorry, what private IOCTLs are you referring to? If you mean private control IDs, then yes, they should go to that header.

[snip]

Quoted 25 lines
Quoted 15 lines
Quoted 15 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +15 -0
+static int mtk_fd_vb2_queue_setup(struct vb2_queue *vq,
+                             unsigned int *num_buffers,
+                             unsigned int *num_planes,
+                             unsigned int sizes[],
+                             struct device *alloc_devs[])
+{
+   struct mtk_fd_ctx *ctx = vb2_get_drv_priv(vq);
+   unsigned int size[2];
+   unsigned int plane;
+
+   switch (vq->type) {
+   case V4L2_BUF_TYPE_META_CAPTURE:
+           size[0] = ctx->dst_fmt.buffersize;
+           break;
+   case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
Quoted 9 lines

drivers/media/platform/mtk-isp/fd/mtk_fd_40.c line 356

Quoted 5 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +5 -0
+           size[0] = ctx->src_fmt.plane_fmt[0].sizeimage;
+           if (*num_planes == 2)
+                   size[1] = ctx->src_fmt.plane_fmt[1].sizeimage;
+           break;
+   }

Is this code above needed? The code below sets sizes[] and it uses a for loop, without opencoded assignment for the second plane.

Quoted 4 lines

Looks like not really useful here,
it should check sizes and num_planes if num_plane not zero, and for V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE, it will at most have 2 planes, maybe no need for loop as well.

Loops generally make the code cleaner and there might be some desire to add support for more formats in the future, e.g. in case a next generation of the hardware shows up.

Quoted 17 lines

I will refine this function as following:
mtk_fd_vb2_queue_setup(...)

{
        struct mtk_fd_ctx *ctx = vb2_get_drv_priv(vq);

        if (*num_planes == 0) {
                if (vq->type == V4L2_BUF_TYPE_META_CAPTURE) {
                        sizes[0] = ctx->dst_fmt.buffersize;
                        *num_planes = 1;
                        return 0;
                } else if (vq->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) {
                        *num_planes = ctx->src_fmt.num_planes;
                        sizes[0] = ctx->src_fmt.plane_fmt[0].sizeimage;
                        if (*num_planes == 2)
                                sizes[1] = ctx->src_fmt.plane_fmt[1].sizeimage;
                        return 0;
                }
Quoted 8 lines
                return -EINVAL;
        }

        /* If num_plane not zero, check the num_plane and sizes*/
        if (vq->type == V4L2_BUF_TYPE_META_CAPTURE) {
                if ((*num_planes == 1) &&
                    (sizes[0] <= ctx->dst_fmt.buffersize))
                        return 0;

nit: The typical convention is to check for problems and return the error code earlier, with the success handled at the end of the block.

Quoted 11 lines
                else
                        return -EINVAL;
        }
        if (vq->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) {
                if ((*num_planes == 1) &&
                    (sizes[0] <= ctx->src_fmt.plane_fmt[0].sizeimage))
                        return 0;
                else if ((*num_planes == 2) &&
                         (sizes[0] <= ctx->src_fmt.plane_fmt[0].sizeimage) &&
                         (sizes[1] <= ctx->src_fmt.plane_fmt[1].sizeimage))
                        return 0;

Wouldn't a loop eliminate the need to if/else if through the various supported cases and duplicate the size checks?

Quoted 6 lines
                else
                        return -EINVAL;

        }
        return 0;
}

How about the following?

mtk_fd_vb2_queue_setup(...)

{
        struct mtk_fd_ctx *ctx = vb2_get_drv_priv(vq);

        if (vq->type == V4L2_BUF_TYPE_META_CAPTURE) {
                if (*num_planes == 0) {
                        *num_planes = 1;
                        sizes[0] = ctx->dst_fmt.buffersize;
                        return 0;
                }

                if (*num_planes != 1 || sizes[0] < ctx->dst_fmt.buffersize)
                                return -EINVAL;

                return 0;
        }

        /* V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE */
        if (*num_planes == 0) {
                        *num_planes = ctx->src_fmt.num_planes;
                        for (i = 0; i < ctx->src_fmt.num_planes; ++i)
                                sizes[i] = ctx->src_fmt.plane_fmt[i].sizeimage;
                        return 0;
        }

        if (*num_planes < ctx->src_fmt.num_planes)
                return -EINVAL;

        for (i = 0; i < ctx->src_fmt.num_planes; ++i)
                if (sizes[i] < ctx->src_fmt.plane_fmt[i].sizeimage)
                        return -EINVAL;

        return 0;
}

Note that it fully separates the code dealing with each queue and thus improves the readability.

In this case, it could actually be beneficial to split the vb2_ops implementation into one that deals only with video_output_mplane and one only with meta_capture. This would allow eliminating the special casing based on vq->type and thus further simplify the code. Not sure if it applies to the other vb2 callbacks, though.

[snip]

Quoted 35 lines
Quoted 24 lines
Quoted 24 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +24 -0
+static void mtk_fd_fill_pixfmt_mp(struct v4l2_pix_format_mplane *dfmt,
+                             const struct v4l2_pix_format_mplane *sfmt)
+{
+   dfmt->field = V4L2_FIELD_NONE;
+   dfmt->colorspace = V4L2_COLORSPACE_BT2020;
+   dfmt->num_planes = sfmt->num_planes;
+   dfmt->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
+   dfmt->quantization = V4L2_QUANTIZATION_DEFAULT;
+   dfmt->xfer_func =
+           V4L2_MAP_XFER_FUNC_DEFAULT(dfmt->colorspace);
+
+   /* Keep user setting as possible */
+   dfmt->width = clamp(dfmt->width,
+                       MTK_FD_OUTPUT_MIN_WIDTH,
+                       MTK_FD_OUTPUT_MAX_WIDTH);
+   dfmt->height = clamp(dfmt->height,
+                        MTK_FD_OUTPUT_MIN_HEIGHT,
+                        MTK_FD_OUTPUT_MAX_HEIGHT);
+
+   if (sfmt->num_planes == 2) {
+           /* NV16M and NV61M has 1 byte per pixel */
+           dfmt->plane_fmt[0].bytesperline = dfmt->width;
+           dfmt->plane_fmt[1].bytesperline = dfmt->width;
+   } else {
Quoted 9 lines

drivers/media/platform/mtk-isp/fd/mtk_fd_40.c line 494

Quoted 6 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +6 -0
+           /* 2 bytes per pixel */
+           dfmt->plane_fmt[0].bytesperline = dfmt->width * 2;
+   }
+
+   dfmt->plane_fmt[0].sizeimage =
+           dfmt->height * dfmt->plane_fmt[0].bytesperline;

Could some of the code above be replaced with v4l2_fill_pixfmt_mp()?

I would like to refine as following

Quoted 7 lines
mtk_fd_fill_pixfmt_mp(...){
        v4l2_fill_pixfmt_mp(dfmt, sfmt->pixelformat, dfmt->width,

dfmt->height);

        dfmt->field = V4L2_FIELD_NONE;
        dfmt->colorspace = V4L2_COLORSPACE_BT2020;
        dfmt->num_planes = sfmt->num_planes;

num_planes should be already filled in by the helper. That actually raises a question on whether there is any need to have sfmt passed to this function at all. Perhaps all we need is the value of pixelformat?

Quoted 6 lines
        dfmt->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
        dfmt->quantization = V4L2_QUANTIZATION_DEFAULT;
        dfmt->xfer_func =
                V4L2_MAP_XFER_FUNC_DEFAULT(dfmt->colorspace);
}

Isn't still a need to clamp() width and height to min/max, though?

[snip]

Quoted 12 lines
Quoted 9 lines

drivers/media/platform/mtk-isp/fd/mtk_fd_40.c line 895

Quoted 5 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +5 -0
+   fd_param.user_param.src_img_fmt =
+           get_fd_img_fmt(ctx->src_fmt.pixelformat);
+   if (ctx->src_fmt.num_planes == 2)
+           fd_param.src_img[1].dma_addr =
+                   vb2_dma_contig_plane_dma_addr(&src_buf->vb2_buf, 1);

nit: Could this be moved above, to be just below src_img[0] initialization, for readability reasons?

Ok, this function will be refined as

static void mtk_fd_device_run(void *priv)

Quoted 14 lines
{
        struct mtk_fd_ctx *ctx = priv;
        struct mtk_fd_dev *fd = ctx->fd_dev;
        struct vb2_v4l2_buffer *src_buf, *dst_buf;
        struct fd_enq_param fd_param;

        src_buf = v4l2_m2m_next_src_buf(ctx->fh.m2m_ctx);
        dst_buf = v4l2_m2m_next_dst_buf(ctx->fh.m2m_ctx);

        fd_param.src_img[0].dma_addr =
                vb2_dma_contig_plane_dma_addr(&src_buf->vb2_buf, 0);
        if (ctx->src_fmt.num_planes == 2)
                fd_param.src_img[1].dma_addr =
                        vb2_dma_contig_plane_dma_addr(&src_buf->vb2_buf, 1);

How about making this a loop in terms of ctx->src_fmt.num_planes?

Quoted 5 lines
        fd_param.user_result.dma_addr =
                vb2_dma_contig_plane_dma_addr(&dst_buf->vb2_buf, 0);
        fd_param.user_param.src_img_fmt =
                get_fd_img_fmt(fd->dev, ctx->src_fmt.pixelformat);
Quoted 8 lines
        mtk_fd_fill_user_param(&fd_param.user_param, &ctx->hdl);

        /* Complete request controls if any */
        v4l2_ctrl_request_complete(src_buf->vb2_buf.req_obj.req, &ctx->hdl);

        fd->output = vb2_plane_vaddr(&dst_buf->vb2_buf, 0);
        mtk_fd_hw_job_exec(fd, &fd_param);
}

Best regards,
Tomasz

#19 lore raw

Re: [RFC PATCH V4 4/4] platform: mtk-isp: Add Mediatek FD driver

Hi Tomasz,

I Appreciate your review comments, here's the reply.

Quoted 54 lines · Tomasz Figa

r

Quoted 7 lines · Jerry-ch Chen

Hi Tomasz,

Quoted 3 lines · Tomasz Figa

Hi Jerry,

On Wed, Dec 04, 2019 at 08:47:32PM +0800, Jerry-ch Chen wrote:

[snip]

Quoted 16 lines
Quoted 6 lines
Quoted 6 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +6 -0
+
+enum face_angle {
+   MTK_FD_FACE_FRONT,
+   MTK_FD_FACE_RIGHT_50,
+   MTK_FD_FACE_LEFT_50,
+   MTK_FD_FACE_RIGHT_90,
Quoted 9 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 61

Quoted 3 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +3 -0
+   MTK_FD_FACE_LEFT_90,
+   MTK_FD_FACE_ANGLE_NUM,
+};

This enum seems to define values for the V4L2_CID_MTK_FD_DETECT_POSE control. Considering that this is an enumeration and the values are actually integers (-90, -50, 0, 50, 90), perhaps this should be an INTEGER_MENU control instead?

Quoted 3 lines

this ioctl let user select multiple face positions(combination of angles and directions) to be detected. so I thought I am not able to use the INTEGER_MENU for this purpose.

Ah, okay, I thought there is only one selection possible.

Quoted 6 lines

A bit-field as following should be used by user.
I consider adding it to uapi.

struct face_direction_def {
__u16 MTK_FD_FACE_DIR_0 : 1,

Quoted 13 lines
        MTK_FD_FACE_DIR_30 : 1,
        MTK_FD_FACE_DIR_60 : 1,
        MTK_FD_FACE_DIR_90 : 1,
        MTK_FD_FACE_DIR_120 : 1,
        MTK_FD_FACE_DIR_150 : 1,
        MTK_FD_FACE_DIR_180 : 1,
        MTK_FD_FACE_DIR_210 : 1,
        MTK_FD_FACE_DIR_240 : 1,
        MTK_FD_FACE_DIR_270 : 1,
        MTK_FD_FACE_DIR_300 : 1,
        MTK_FD_FACE_DIR_330 : 1,
        : 4;
};
Quoted 4 lines

Note that bit fields are not recommended in UAPI because of not being well specified by the language. Instead bits should be defined using macros with explicit masks or sometimes enums.

Ok, I'll define them in macro with masks.

Quoted 46 lines
Quoted 11 lines

User can also select some face directions of each face angle in one ioctl, for example:

/*
* u16 face_directions[MTK_FD_FACE_ANGLE_NUM] = {0}; *
* face_directions[MTK_FD_FACE_FRONT] = 0x7; //angle:0, dir:0,30,60
* face_directions[MTK_FACE_RIGHT_50] = 0x2; //angle:50, dir:30
*
*/

Makes sense, thanks.

Quoted 19 lines
Quoted 6 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 66

Quoted 6 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +5 -0
+
+struct fd_buffer {
+   __u32 scp_addr; /* used by SCP */
+   __u32 dma_addr; /* used by DMA HW */
+} __packed;

fd buffer is used for scp ipi

Quoted 4 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 70

Quoted 4 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +4 -0
+
+struct fd_face_result {
+   char data[16];
+};

fd_face_result is used for user, so it should be moved to include/uapi/linux.
In fact, it has bit-field definition for user, so I would like to define it in include/uapi/linux as following:

struct fd_face_result {
__u64 face_idx : 12,

Quoted 12 lines
        type : 1,
        x0 : 10,
        y0 : 10,
        x1 : 10,
        y1 : 10,
        fcv1 : 11;
  __u64 fcv2 : 7,
        rip_dir : 4,
        rop_dir : 3,
        det_size : 5;
};
Quoted 4 lines

Indeed this should be defined, but as per my comment above, please avoid using the bitfield construct and define shifts and masks instead.

Ok, I'll fix it.

Quoted 15 lines
Quoted 15 lines
Quoted 9 lines

drivers/media/platform/mtk-isp/fd/mtk_fd.h line 74

Quoted 5 lines
drivers/media/platform/mtk-isp/fd/mtk_fd.h new file +4 -0
+
+struct fd_user_output {
+   struct fd_face_result results[MTK_FD_MAX_RESULT_NUM];
+   __u16 number;

Is this perhaps the number of results? If so, would num_results be a better name?

yes, fixed.

Quoted 5 lines
Quoted 1 line

+};

Since this struct is the meta buffer format, it is a part of the userspace interface and should be defined in a header under include/uapi/linux/.

Quoted 7 lines
Quoted 4 lines

Ok, I will create include/uapi/linux/mtk_fd_40.h which suppose to include structures that userspace will use. should the private IOCTLs be placed in it together?

Sorry, what private IOCTLs are you referring to? If you mean private control IDs, then yes, they should go to that header.

yes, the IDs, sorry for the wrong expression.

Quoted 27 lines

[snip]

Quoted 25 lines
Quoted 24 lines
Quoted 10 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +10 -0
+static int mtk_fd_vb2_queue_setup(struct vb2_queue *vq,
+                             unsigned int *num_buffers,
+                             unsigned int *num_planes,
+                             unsigned int sizes[],
+                             struct device *alloc_devs[])
+{
+   struct mtk_fd_ctx *ctx = vb2_get_drv_priv(vq);
+   unsigned int size[2];
+   unsigned int plane;
+

drivers/media/platform/mtk-isp/fd/mtk_fd_40.c line 356

Quoted 10 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +10 -0
+   switch (vq->type) {
+   case V4L2_BUF_TYPE_META_CAPTURE:
+           size[0] = ctx->dst_fmt.buffersize;
+           break;
+   case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
+           size[0] = ctx->src_fmt.plane_fmt[0].sizeimage;
+           if (*num_planes == 2)
+                   size[1] = ctx->src_fmt.plane_fmt[1].sizeimage;
+           break;
+   }

Is this code above needed? The code below sets sizes[] and it uses a for loop, without opencoded assignment for the second plane.

Quoted 9 lines
Quoted 4 lines

Looks like not really useful here,
it should check sizes and num_planes if num_plane not zero, and for V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE, it will at most have 2 planes, maybe no need for loop as well.

Loops generally make the code cleaner and there might be some desire to add support for more formats in the future, e.g. in case a next generation of the hardware shows up.

Ok, got it.

Quoted 20 lines
Quoted 20 lines

I will refine this function as following:
mtk_fd_vb2_queue_setup(...)

{
        struct mtk_fd_ctx *ctx = vb2_get_drv_priv(vq);

        if (*num_planes == 0) {
                if (vq->type == V4L2_BUF_TYPE_META_CAPTURE) {
                        sizes[0] = ctx->dst_fmt.buffersize;
                        *num_planes = 1;
                        return 0;
                } else if (vq->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) {
                        *num_planes = ctx->src_fmt.num_planes;
                        sizes[0] = ctx->src_fmt.plane_fmt[0].sizeimage;
                        if (*num_planes == 2)
                                sizes[1] = ctx->src_fmt.plane_fmt[1].sizeimage;
                        return 0;
                }
                return -EINVAL;
        }
Quoted 9 lines
Quoted 5 lines
        /* If num_plane not zero, check the num_plane and sizes*/
        if (vq->type == V4L2_BUF_TYPE_META_CAPTURE) {
                if ((*num_planes == 1) &&
                    (sizes[0] <= ctx->dst_fmt.buffersize))
                        return 0;

nit: The typical convention is to check for problems and return the error code earlier, with the success handled at the end of the block.

Got it.

Quoted 59 lines
Quoted 11 lines
                else
                        return -EINVAL;
        }
        if (vq->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) {
                if ((*num_planes == 1) &&
                    (sizes[0] <= ctx->src_fmt.plane_fmt[0].sizeimage))
                        return 0;
                else if ((*num_planes == 2) &&
                         (sizes[0] <= ctx->src_fmt.plane_fmt[0].sizeimage) &&
                         (sizes[1] <= ctx->src_fmt.plane_fmt[1].sizeimage))
                        return 0;

Wouldn't a loop eliminate the need to if/else if through the various supported cases and duplicate the size checks?

Quoted 6 lines
                else
                        return -EINVAL;

        }
        return 0;
}

How about the following?

mtk_fd_vb2_queue_setup(...)

{
        struct mtk_fd_ctx *ctx = vb2_get_drv_priv(vq);

        if (vq->type == V4L2_BUF_TYPE_META_CAPTURE) {
                if (*num_planes == 0) {
                        *num_planes = 1;
                        sizes[0] = ctx->dst_fmt.buffersize;
                        return 0;
                }

                if (*num_planes != 1 || sizes[0] < ctx->dst_fmt.buffersize)
                                return -EINVAL;

                return 0;
        }

        /* V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE */
        if (*num_planes == 0) {
                        *num_planes = ctx->src_fmt.num_planes;
                        for (i = 0; i < ctx->src_fmt.num_planes; ++i)
                                sizes[i] = ctx->src_fmt.plane_fmt[i].sizeimage;
                        return 0;
        }

        if (*num_planes < ctx->src_fmt.num_planes)
                return -EINVAL;

        for (i = 0; i < ctx->src_fmt.num_planes; ++i)
                if (sizes[i] < ctx->src_fmt.plane_fmt[i].sizeimage)
                        return -EINVAL;

        return 0;
}
Quoted 9 lines

Note that it fully separates the code dealing with each queue and thus improves the readability.

In this case, it could actually be beneficial to split the vb2_ops implementation into one that deals only with video_output_mplane and one only with meta_capture. This would allow eliminating the special casing based on vq->type and thus further simplify the code. Not sure if it applies to the other vb2 callbacks, though.

Got it, thanks for the comments.

Quoted 40 lines

[snip]

Quoted 39 lines
Quoted 24 lines
Quoted 24 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +24 -0
+static void mtk_fd_fill_pixfmt_mp(struct v4l2_pix_format_mplane *dfmt,
+                             const struct v4l2_pix_format_mplane *sfmt)
+{
+   dfmt->field = V4L2_FIELD_NONE;
+   dfmt->colorspace = V4L2_COLORSPACE_BT2020;
+   dfmt->num_planes = sfmt->num_planes;
+   dfmt->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
+   dfmt->quantization = V4L2_QUANTIZATION_DEFAULT;
+   dfmt->xfer_func =
+           V4L2_MAP_XFER_FUNC_DEFAULT(dfmt->colorspace);
+
+   /* Keep user setting as possible */
+   dfmt->width = clamp(dfmt->width,
+                       MTK_FD_OUTPUT_MIN_WIDTH,
+                       MTK_FD_OUTPUT_MAX_WIDTH);
+   dfmt->height = clamp(dfmt->height,
+                        MTK_FD_OUTPUT_MIN_HEIGHT,
+                        MTK_FD_OUTPUT_MAX_HEIGHT);
+
+   if (sfmt->num_planes == 2) {
+           /* NV16M and NV61M has 1 byte per pixel */
+           dfmt->plane_fmt[0].bytesperline = dfmt->width;
+           dfmt->plane_fmt[1].bytesperline = dfmt->width;
+   } else {
Quoted 9 lines

drivers/media/platform/mtk-isp/fd/mtk_fd_40.c line 494

Quoted 6 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +6 -0
+           /* 2 bytes per pixel */
+           dfmt->plane_fmt[0].bytesperline = dfmt->width * 2;
+   }
+
+   dfmt->plane_fmt[0].sizeimage =
+           dfmt->height * dfmt->plane_fmt[0].bytesperline;

Could some of the code above be replaced with v4l2_fill_pixfmt_mp()?

I would like to refine as following

mtk_fd_fill_pixfmt_mp(...){
        v4l2_fill_pixfmt_mp(dfmt, sfmt->pixelformat, dfmt->width,

dfmt->height);

Quoted 8 lines
Quoted 3 lines
        dfmt->field = V4L2_FIELD_NONE;
        dfmt->colorspace = V4L2_COLORSPACE_BT2020;
        dfmt->num_planes = sfmt->num_planes;

num_planes should be already filled in by the helper. That actually raises a question on whether there is any need to have sfmt passed to this function at all. Perhaps all we need is the value of pixelformat?

Yes, I'll replace sfmt with u32 pixfmt.

Quoted 8 lines
Quoted 6 lines
        dfmt->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
        dfmt->quantization = V4L2_QUANTIZATION_DEFAULT;
        dfmt->xfer_func =
                V4L2_MAP_XFER_FUNC_DEFAULT(dfmt->colorspace);
}

Isn't still a need to clamp() width and height to min/max, though?

Yes, I'll add them back.

This function will be refined as :

static void mtk_fd_fill_pixfmt_mp(struct v4l2_pix_format_mplane *dfmt,

                                  u32 pixfmt)
{
        v4l2_fill_pixfmt_mp(dfmt, pixfmt, dfmt->width, dfmt->height);

        dfmt->field = V4L2_FIELD_NONE;
        dfmt->colorspace = V4L2_COLORSPACE_BT2020;
        dfmt->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
        dfmt->quantization = V4L2_QUANTIZATION_DEFAULT;
        dfmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(dfmt->colorspace);

        /* Keep user setting as possible */
        dfmt->width = clamp(dfmt->width,
                            MTK_FD_OUTPUT_MIN_WIDTH,
                            MTK_FD_OUTPUT_MAX_WIDTH);
        dfmt->height = clamp(dfmt->height,
                             MTK_FD_OUTPUT_MIN_HEIGHT,
                             MTK_FD_OUTPUT_MAX_HEIGHT);
}
Quoted 23 lines

[snip]

Quoted 21 lines
Quoted 9 lines

drivers/media/platform/mtk-isp/fd/mtk_fd_40.c line 895

Quoted 5 lines
drivers/media/platform/mtk-isp/fd/mtk_fd_40.c new file +5 -0
+   fd_param.user_param.src_img_fmt =
+           get_fd_img_fmt(ctx->src_fmt.pixelformat);
+   if (ctx->src_fmt.num_planes == 2)
+           fd_param.src_img[1].dma_addr =
+                   vb2_dma_contig_plane_dma_addr(&src_buf->vb2_buf, 1);

nit: Could this be moved above, to be just below src_img[0] initialization, for readability reasons?

Ok, this function will be refined as

static void mtk_fd_device_run(void *priv)

{
        struct mtk_fd_ctx *ctx = priv;
        struct mtk_fd_dev *fd = ctx->fd_dev;
        struct vb2_v4l2_buffer *src_buf, *dst_buf;
        struct fd_enq_param fd_param;

        src_buf = v4l2_m2m_next_src_buf(ctx->fh.m2m_ctx);
        dst_buf = v4l2_m2m_next_dst_buf(ctx->fh.m2m_ctx);
Quoted 8 lines
Quoted 5 lines
        fd_param.src_img[0].dma_addr =
                vb2_dma_contig_plane_dma_addr(&src_buf->vb2_buf, 0);
        if (ctx->src_fmt.num_planes == 2)
                fd_param.src_img[1].dma_addr =
                        vb2_dma_contig_plane_dma_addr(&src_buf->vb2_buf, 1);

How about making this a loop in terms of ctx->src_fmt.num_planes?

yes, it will be refined as following, I use the src_vb2_buf to reduce the length for fitting 80 columns

src_vb2_buf = &src_buf->vb2_buf;
dst_vb2_buf = &dst_buf->vb2_buf;

for (i = 0; i < ctx->src_fmt.num_planes; i++)

	fd_param.src_img[i].dma_addr =
		vb2_dma_contig_plane_dma_addr(src_vb2_buf,i);

fd_param.user_result.dma_addr =

	vb2_dma_contig_plane_dma_addr(dst_vb2_buf, 0);
Quoted 10 lines
Quoted 10 lines
        fd_param.user_result.dma_addr =
                vb2_dma_contig_plane_dma_addr(&dst_buf->vb2_buf, 0);
        fd_param.user_param.src_img_fmt =
                get_fd_img_fmt(fd->dev, ctx->src_fmt.pixelformat);

        mtk_fd_fill_user_param(&fd_param.user_param, &ctx->hdl);

        /* Complete request controls if any */
        v4l2_ctrl_request_complete(src_buf->vb2_buf.req_obj.req, &ctx->hdl);
Quoted 6 lines
Quoted 3 lines
        fd->output = vb2_plane_vaddr(&dst_buf->vb2_buf, 0);
        mtk_fd_hw_job_exec(fd, &fd_param);
}

Best regards,
Tomasz

Thanks and Best regards,
Jerry

#20 lore raw

Re: [RFC PATCH V4 4/4] platform: mtk-isp: Add Mediatek FD driver

Quoted 17 lines · Jerry-ch Chen

Hi Tomasz,

I Appreciate your review comments, here's the reply.

Quoted 11 lines · Tomasz Figa

r

Quoted 7 lines · Jerry-ch Chen

Hi Tomasz,

Quoted 3 lines · Tomasz Figa

Hi Jerry,

On Wed, Dec 04, 2019 at 08:47:32PM +0800, Jerry-ch Chen wrote:

[snip]

Quoted 16 lines
Quoted 1 line

Isn't still a need to clamp() width and height to min/max, though?

Yes, I'll add them back.

This function will be refined as :

static void mtk_fd_fill_pixfmt_mp(struct v4l2_pix_format_mplane *dfmt,

                                  u32 pixfmt)
{
        v4l2_fill_pixfmt_mp(dfmt, pixfmt, dfmt->width, dfmt->height);

        dfmt->field = V4L2_FIELD_NONE;
        dfmt->colorspace = V4L2_COLORSPACE_BT2020;
        dfmt->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
        dfmt->quantization = V4L2_QUANTIZATION_DEFAULT;
        dfmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(dfmt->colorspace);
Quoted 7 lines
        /* Keep user setting as possible */
        dfmt->width = clamp(dfmt->width,
                            MTK_FD_OUTPUT_MIN_WIDTH,
                            MTK_FD_OUTPUT_MAX_WIDTH);
        dfmt->height = clamp(dfmt->height,
                             MTK_FD_OUTPUT_MIN_HEIGHT,
                             MTK_FD_OUTPUT_MAX_HEIGHT);

Note that this would cause the other fields of dfmt to be inconsistent with width and height. The correct way to do this would be to first clamp and then call v4l2_fill_pixfmt_mp().

Best regards,
Tomasz

#21 lore raw

Re: [RFC PATCH V4 4/4] platform: mtk-isp: Add Mediatek FD driver

Quoted 44 lines · Tomasz Figa
Quoted 17 lines · Jerry-ch Chen

Hi Tomasz,

I Appreciate your review comments, here's the reply.

Quoted 11 lines · Tomasz Figa

r

Quoted 7 lines · Jerry-ch Chen

Hi Tomasz,

Quoted 3 lines · Tomasz Figa

Hi Jerry,

On Wed, Dec 04, 2019 at 08:47:32PM +0800, Jerry-ch Chen wrote:

[snip]

Quoted 16 lines
Quoted 1 line

Isn't still a need to clamp() width and height to min/max, though?

Yes, I'll add them back.

This function will be refined as :

static void mtk_fd_fill_pixfmt_mp(struct v4l2_pix_format_mplane *dfmt,

                                  u32 pixfmt)
{
        v4l2_fill_pixfmt_mp(dfmt, pixfmt, dfmt->width, dfmt->height);

        dfmt->field = V4L2_FIELD_NONE;
        dfmt->colorspace = V4L2_COLORSPACE_BT2020;
        dfmt->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
        dfmt->quantization = V4L2_QUANTIZATION_DEFAULT;
        dfmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(dfmt->colorspace);
Quoted 7 lines
        /* Keep user setting as possible */
        dfmt->width = clamp(dfmt->width,
                            MTK_FD_OUTPUT_MIN_WIDTH,
                            MTK_FD_OUTPUT_MAX_WIDTH);
        dfmt->height = clamp(dfmt->height,
                             MTK_FD_OUTPUT_MIN_HEIGHT,
                             MTK_FD_OUTPUT_MAX_HEIGHT);
Quoted 4 lines

Note that this would cause the other fields of dfmt to be inconsistent with width and height. The correct way to do this would be to first clamp and then call v4l2_fill_pixfmt_mp().

Ok, I will fix it.

Thanks and Best regards,
Jerry

Quoted 2 lines

Best regards,
Tomasz

#22 lore raw
In reply to Jerry-ch Chen

Hi Laurent, Tomasz, Matthias,

gentle ping for this patch set,
If no new comments, I would like to send a newer version.

Thanks and Best Regards,
Jerry

Quoted 247 lines · Jerry-ch Chen

Hello,

This RFC patch series is adding Face Detection (FD) driver on Mediatek mt8183 SoC. It belongs to the first Mediatek's camera driver series based on V4L2 and media controller framework. I posted the main part of the FD driver as RFC to discuss first and would like some review comments.

==============
Introduction
==============

Face Detection (FD) unit provides hardware accelerated face detection feature. It can detect different sizes of faces in a given image.

The driver is implemented as a normal V4L2 memory-to-memory device and supports V4L2 controls for detection settings. It has two buffer queues.

1. Video output buffer: RAW image for face detection.

2. Meta capture buffer: Result of the detected faces.

==================
Changes in v4
==================

RFC v4 includes the following modification:
1. Includes v4l2-mem2mem changes: add v4l2_m2m_suspend, v4l2_m2m_resume

2. Move FD V4L2 control ids back inside FD driver

3. Depend on newer SCP driver api

4. Add exit hw flow for FD driver

5. Add hardware timeout handling in the FD firmware

6. Move allocation of dma buffer from connect / disconnect to driver probe / remove

Todo:
- Add v4l2 control menus for private mtk fd control
==================
Changes in v3
==================

RFC v3 includes the following modification:
1. Adjust the private control ids and place them in include/uapi/linux/mtk-fd-v4l2-controls.h

2. Merge struct mtk_fd_hw info struct mtk_fd_dev

3. Define FD meta capture buffer in include/uapi/linux/videodev2.h

4. Remove the usage of get_reserved_memory by scp driver, use dma_alloc api instead

Todo:
- Add v4l2 control menus for private mtk fd control
- Refine the job finish flow when system suspend
==================
Changes in v2
==================

RFC v2 includes the following modification:
1. Implement FD as a V4L2 mem2mem driver

2. Replace meta input with V4L2 controls

==================
Changes in v1
==================

RFC v1 includes the following modification:
1. Uses Request API instead of FD's buffer collection design

2. removed unnecessary abstraction structurally, including mtk_fd_ctx and related ops

3. removed the fd_smem node from device tree

4. Fixed the common issues Tomasz commented on Mediatek ISP Pass 1's RFC v0 patch series

==================
Dependent patch
==================

FD driver depends on SCP driver. The patches are as following:

[1]. Add support for mt8183 SCP
https://patchwork.kernel.org/cover/1152350/

==================
Compliance test
==================

* Version: https://git.linuxtv.org/v4l-utils.git/commit/?id=b16f9e945d74aa5
* Note: Some failures are caused by the implementation of FD driver, whic is a m2m device with VIDEO_OUT and META_CAPTURE queues, therefore we can't set V4L2_CAP_VIDEO_M2M in device capability, and fail in some non-m2m v4l2 test cases.
* Test command: v4l2-compliance -m 2
* test output:

v4l2-compliance SHA: not available, 32 bits

Compliance test for mtk-fd-4.0 device /dev/media2:

Media Driver Info:

        Driver name      : mtk-fd-4.0
        Model            : mtk-fd-4.0
        Serial           :
        Bus info         : platform:1502b000.fd
        Media version    : 4.19.84
        Hardware revision: 0x00000000 (0)
        Driver version   : 4.19.84

Required ioctls:
test MEDIA_IOC_DEVICE_INFO: OK

Allow for multiple opens:
test second /dev/media2 open: OK
test MEDIA_IOC_DEVICE_INFO: OK
test for unlimited opens: OK

Media Controller ioctls:
test MEDIA_IOC_G_TOPOLOGY: OK
Entities: 3 Interfaces: 1 Pads: 4 Links: 4 test MEDIA_IOC_ENUM_ENTITIES/LINKS: OK test MEDIA_IOC_SETUP_LINK: OK

Total for mtk-fd-4.0 device /dev/media2: 7, Succeeded: 7, Failed: 0, Warnings: 0
--------------------------------------------------------------------------------
Compliance test for mtk-fd-4.0 device /dev/video32:

Driver Info:

        Driver name      : mtk-fd-4.0
        Card type        : mtk-fd-4.0
        Bus info         : platform:1502b000.fd
        Driver version   : 4.19.84
        Capabilities     : 0x84a02000

Video Output Multiplanar
Metadata Capture
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04a02000
Video Output Multiplanar
Metadata Capture
Streaming
Extended Pix Format
Media Driver Info:

        Driver name      : mtk-fd-4.0
        Model            : mtk-fd-4.0
        Serial           :
        Bus info         : platform:1502b000.fd
        Media version    : 4.19.84
        Hardware revision: 0x00000000 (0)
        Driver version   : 4.19.84

Interface Info:
ID : 0x0300000c
Type : V4L Video
Entity Info:

        ID               : 0x00000001 (1)
        Name             : mtk-fd-4.0-source
        Function         : V4L2 I/O
        Pad 0x01000002   : 0: Source
          Link 0x02000008: to remote pad 0x1000005 of entity 'mtk-fd-4.0-proc': Data, Enabled, Immutable

Required ioctls:
test MC information (see 'Media Driver Info' above): OK
fail: v4l2-compliance.cpp(668): dcaps & output_caps
test VIDIOC_QUERYCAP: FAIL

Allow for multiple opens:
test second /dev/video32 open: OK
fail: v4l2-compliance.cpp(668): dcaps & output_caps
test VIDIOC_QUERYCAP: FAIL
test VIDIOC_G/S_PRIORITY: OK
test for unlimited opens: OK

Debug ioctls:
test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported) test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported) test VIDIOC_ENUMAUDIO: OK (Not Supported)
test VIDIOC_G/S/ENUMINPUT: OK (Not Supported) test VIDIOC_G/S_AUDIO: OK (Not Supported)
Inputs: 0 Audio Inputs: 0 Tuners: 0

Output ioctls:
test VIDIOC_G/S_MODULATOR: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_ENUMAUDOUT: OK (Not Supported) test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported) test VIDIOC_G/S_AUDOUT: OK (Not Supported)
Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported) test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported) test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported) test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls:
test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK test VIDIOC_QUERYCTRL: OK
test VIDIOC_G/S_CTRL: OK
test VIDIOC_G/S/TRY_EXT_CTRLS: OK test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK test VIDIOC_G/S_JPEGCOMP: OK (Not Supported) Standard Controls: 1 Private Controls: 6

Format ioctls:
test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
test VIDIOC_G/S_PARM: OK (Not Supported)
test VIDIOC_G_FBUF: OK (Not Supported)
fail: v4l2-test-formats.cpp(457): pixelformat 00000000 () for buftype 10 not reported by ENUM_FMT
test VIDIOC_G_FMT: FAIL
fail: v4l2-test-formats.cpp(457): pixelformat 00000000 () for buftype 10 not reported by ENUM_FMT
test VIDIOC_TRY_FMT: FAIL
fail: v4l2-test-formats.cpp(457): pixelformat ffffffff (-BE) for buftype 10 not reported by ENUM_FMT
test VIDIOC_S_FMT: FAIL
test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
test Cropping: OK (Not Supported)
test Composing: OK (Not Supported)
test Scaling: OK (Not Supported)

Codec ioctls:
test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported) test VIDIOC_G_ENC_INDEX: OK (Not Supported) test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls:
fail: v4l2-test-buffers.cpp(667): q2.reqbufs(node->node2, 1) != EBUSY
test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: FAIL
test VIDIOC_EXPBUF: OK
test Requests: OK

Total for mtk-fd-4.0 device /dev/video32: 45, Succeeded: 39, Failed: 6, Warnings: 0

Grand Total for mtk-fd-4.0 device /dev/media2: 52, Succeeded: 46, Failed: 6, Warnings: 0

#23 lore raw

Hi Jerry,

Quoted 6 lines · Jerry-ch Chen

Hi Laurent, Tomasz, Matthias,

gentle ping for this patch set,
If no new comments, I would like to send a newer version.

Sorry, I still haven't had a chance to look at the series, so feel free to send a new version and I will take a look at the new one.

Best regards,
Tomasz

Quoted 252 lines

Thanks and Best Regards,
Jerry

Quoted 247 lines · Jerry-ch Chen

Hello,

This RFC patch series is adding Face Detection (FD) driver on Mediatek mt8183 SoC. It belongs to the first Mediatek's camera driver series based on V4L2 and media controller framework. I posted the main part of the FD driver as RFC to discuss first and would like some review comments.

==============
Introduction
==============

Face Detection (FD) unit provides hardware accelerated face detection feature. It can detect different sizes of faces in a given image.

The driver is implemented as a normal V4L2 memory-to-memory device and supports V4L2 controls for detection settings. It has two buffer queues.

1. Video output buffer: RAW image for face detection.

2. Meta capture buffer: Result of the detected faces.

==================
Changes in v4
==================

RFC v4 includes the following modification:
1. Includes v4l2-mem2mem changes: add v4l2_m2m_suspend, v4l2_m2m_resume

2. Move FD V4L2 control ids back inside FD driver

3. Depend on newer SCP driver api

4. Add exit hw flow for FD driver

5. Add hardware timeout handling in the FD firmware

6. Move allocation of dma buffer from connect / disconnect to driver probe / remove

Todo:
- Add v4l2 control menus for private mtk fd control
==================
Changes in v3
==================

RFC v3 includes the following modification:
1. Adjust the private control ids and place them in include/uapi/linux/mtk-fd-v4l2-controls.h

2. Merge struct mtk_fd_hw info struct mtk_fd_dev

3. Define FD meta capture buffer in include/uapi/linux/videodev2.h

4. Remove the usage of get_reserved_memory by scp driver, use dma_alloc api instead

Todo:
- Add v4l2 control menus for private mtk fd control
- Refine the job finish flow when system suspend
==================
Changes in v2
==================

RFC v2 includes the following modification:
1. Implement FD as a V4L2 mem2mem driver

2. Replace meta input with V4L2 controls

==================
Changes in v1
==================

RFC v1 includes the following modification:
1. Uses Request API instead of FD's buffer collection design

2. removed unnecessary abstraction structurally, including mtk_fd_ctx and related ops

3. removed the fd_smem node from device tree

4. Fixed the common issues Tomasz commented on Mediatek ISP Pass 1's RFC v0 patch series

==================
Dependent patch
==================

FD driver depends on SCP driver. The patches are as following:

[1]. Add support for mt8183 SCP
https://patchwork.kernel.org/cover/1152350/

==================
Compliance test
==================

* Version: https://git.linuxtv.org/v4l-utils.git/commit/?id=b16f9e945d74aa5
* Note: Some failures are caused by the implementation of FD driver, whic is a m2m device with VIDEO_OUT and META_CAPTURE queues, therefore we can't set V4L2_CAP_VIDEO_M2M in device capability, and fail in some non-m2m v4l2 test cases.
* Test command: v4l2-compliance -m 2
* test output:

v4l2-compliance SHA: not available, 32 bits

Compliance test for mtk-fd-4.0 device /dev/media2:

Media Driver Info:

        Driver name      : mtk-fd-4.0
        Model            : mtk-fd-4.0
        Serial           :
        Bus info         : platform:1502b000.fd
        Media version    : 4.19.84
        Hardware revision: 0x00000000 (0)
        Driver version   : 4.19.84

Required ioctls:
test MEDIA_IOC_DEVICE_INFO: OK

Allow for multiple opens:
test second /dev/media2 open: OK
test MEDIA_IOC_DEVICE_INFO: OK
test for unlimited opens: OK

Media Controller ioctls:
test MEDIA_IOC_G_TOPOLOGY: OK
Entities: 3 Interfaces: 1 Pads: 4 Links: 4 test MEDIA_IOC_ENUM_ENTITIES/LINKS: OK test MEDIA_IOC_SETUP_LINK: OK

Total for mtk-fd-4.0 device /dev/media2: 7, Succeeded: 7, Failed: 0, Warnings: 0
--------------------------------------------------------------------------------
Compliance test for mtk-fd-4.0 device /dev/video32:

Driver Info:

        Driver name      : mtk-fd-4.0
        Card type        : mtk-fd-4.0
        Bus info         : platform:1502b000.fd
        Driver version   : 4.19.84
        Capabilities     : 0x84a02000

Video Output Multiplanar
Metadata Capture
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04a02000
Video Output Multiplanar
Metadata Capture
Streaming
Extended Pix Format
Media Driver Info:

        Driver name      : mtk-fd-4.0
        Model            : mtk-fd-4.0
        Serial           :
        Bus info         : platform:1502b000.fd
        Media version    : 4.19.84
        Hardware revision: 0x00000000 (0)
        Driver version   : 4.19.84

Interface Info:
ID : 0x0300000c
Type : V4L Video
Entity Info:

        ID               : 0x00000001 (1)
        Name             : mtk-fd-4.0-source
        Function         : V4L2 I/O
        Pad 0x01000002   : 0: Source
          Link 0x02000008: to remote pad 0x1000005 of entity 'mtk-fd-4.0-proc': Data, Enabled, Immutable

Required ioctls:
test MC information (see 'Media Driver Info' above): OK
fail: v4l2-compliance.cpp(668): dcaps & output_caps
test VIDIOC_QUERYCAP: FAIL

Allow for multiple opens:
test second /dev/video32 open: OK
fail: v4l2-compliance.cpp(668): dcaps & output_caps
test VIDIOC_QUERYCAP: FAIL
test VIDIOC_G/S_PRIORITY: OK
test for unlimited opens: OK

Debug ioctls:
test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported) test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported) test VIDIOC_ENUMAUDIO: OK (Not Supported)
test VIDIOC_G/S/ENUMINPUT: OK (Not Supported) test VIDIOC_G/S_AUDIO: OK (Not Supported)
Inputs: 0 Audio Inputs: 0 Tuners: 0

Output ioctls:
test VIDIOC_G/S_MODULATOR: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_ENUMAUDOUT: OK (Not Supported) test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported) test VIDIOC_G/S_AUDOUT: OK (Not Supported)
Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported) test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported) test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported) test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls:
test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK test VIDIOC_QUERYCTRL: OK
test VIDIOC_G/S_CTRL: OK
test VIDIOC_G/S/TRY_EXT_CTRLS: OK test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK test VIDIOC_G/S_JPEGCOMP: OK (Not Supported) Standard Controls: 1 Private Controls: 6

Format ioctls:
test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
test VIDIOC_G/S_PARM: OK (Not Supported)
test VIDIOC_G_FBUF: OK (Not Supported)
fail: v4l2-test-formats.cpp(457): pixelformat 00000000 () for buftype 10 not reported by ENUM_FMT
test VIDIOC_G_FMT: FAIL
fail: v4l2-test-formats.cpp(457): pixelformat 00000000 () for buftype 10 not reported by ENUM_FMT
test VIDIOC_TRY_FMT: FAIL
fail: v4l2-test-formats.cpp(457): pixelformat ffffffff (-BE) for buftype 10 not reported by ENUM_FMT
test VIDIOC_S_FMT: FAIL
test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
test Cropping: OK (Not Supported)
test Composing: OK (Not Supported)
test Scaling: OK (Not Supported)

Codec ioctls:
test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported) test VIDIOC_G_ENC_INDEX: OK (Not Supported) test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls:
fail: v4l2-test-buffers.cpp(667): q2.reqbufs(node->node2, 1) != EBUSY
test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: FAIL
test VIDIOC_EXPBUF: OK
test Requests: OK

Total for mtk-fd-4.0 device /dev/video32: 45, Succeeded: 39, Failed: 6, Warnings: 0

Grand Total for mtk-fd-4.0 device /dev/media2: 52, Succeeded: 46, Failed: 6, Warnings: 0

#24 lore raw

Hi Jerry,

Quoted 4 lines · Tomasz Figa

Hi Jerry,

Quoted 1 line · Jerry-ch Chen
Quoted 9 lines
Quoted 5 lines

Hi Laurent, Tomasz, Matthias,

gentle ping for this patch set,
If no new comments, I would like to send a newer version.

Sorry, I still haven't had a chance to look at the series, so feel free to send a new version and I will take a look at the new one.

Finally found some time to review the series. Again sorry for the delay and thanks for your patience.

Some general comments:
1) The metadata format FourCC should be added in a separate patch, together with documentation for it.
2) Control IDs, structs used by the userspace, etc. should be defined in a header under include/uapi/linux.

Please also check my replies to particular patches for further comments.

Best regards,
Tomasz

#25 lore raw

Hi Tomasz,

Quoted 26 lines · Tomasz Figa

Hi Jerry,

Quoted 4 lines · Tomasz Figa

Hi Jerry,

Quoted 1 line · Jerry-ch Chen
Quoted 9 lines
Quoted 5 lines

Hi Laurent, Tomasz, Matthias,

gentle ping for this patch set,
If no new comments, I would like to send a newer version.

Sorry, I still haven't had a chance to look at the series, so feel free to send a new version and I will take a look at the new one.

Finally found some time to review the series. Again sorry for the delay and thanks for your patience.

Some general comments:
1) The metadata format FourCC should be added in a separate patch, together with documentation for it.
2) Control IDs, structs used by the userspace, etc. should be defined in a header under include/uapi/linux.

Quoted 4 lines

Please also check my replies to particular patches for further comments.

Best regards,
Tomasz

Appreciate for your reply,

So far, I've locally created an uapi header:
include/uapi/linux/mtk_fd_40.h
which provides some values, control ids, and the definitions of structures that would be needed by user of mtk_fd_40 driver. In addition, I also provide a MACRO as example in comments that can extract the struct member with bit length and offset definitions(eliminate the bit-fields).

Also, I would like to rename struct fd_user_output with struct mtk_fd_hw_result. I worry fd_user_output would be a confusing name. I will add them in a separate patch in next version.

I am still working on the documentation, which might be Documentation/media/uapi/v4l/pixfmt-meta-mtk-fd-40.rst. Refering the other pixfmt-*.rst files, I will try to provide the flat-table of the metadata with the structure of the mtk_fd_hw_result.

I am confusing that should I remain the name with -40 in the tail of rst file?
Since I think the layout of metadata might be different in the future mtk fd drivers. Maybe they create a new one or should they update the rst file when they are upstreaming?

Other comments are almost fixed, I will inform you by this mail thread as soon as I finish the documentation of fd metadata format.

Sorry for the late response,
Thanks and Best Regards,
Jerry

#26 lore raw

Hi Jerry,

Quoted 44 lines · Jerry-ch Chen

Hi Tomasz,

Quoted 26 lines · Tomasz Figa

Hi Jerry,

Quoted 4 lines · Tomasz Figa

Hi Jerry,

Quoted 1 line · Jerry-ch Chen
Quoted 9 lines
Quoted 5 lines

Hi Laurent, Tomasz, Matthias,

gentle ping for this patch set,
If no new comments, I would like to send a newer version.

Sorry, I still haven't had a chance to look at the series, so feel free to send a new version and I will take a look at the new one.

Finally found some time to review the series. Again sorry for the delay and thanks for your patience.

Some general comments:
1) The metadata format FourCC should be added in a separate patch, together with documentation for it.
2) Control IDs, structs used by the userspace, etc. should be defined in a header under include/uapi/linux.

Quoted 4 lines

Please also check my replies to particular patches for further comments.

Best regards,
Tomasz

Appreciate for your reply,

So far, I've locally created an uapi header:
include/uapi/linux/mtk_fd_40.h
which provides some values, control ids, and the definitions of structures that would be needed by user of mtk_fd_40 driver. In addition, I also provide a MACRO as example in comments that can extract the struct member with bit length and offset definitions(eliminate the bit-fields).

Quoted 2 lines

Also, I would like to rename struct fd_user_output with struct mtk_fd_hw_result. I worry fd_user_output would be a confusing name.

The change sounds good to me.

Quoted 2 lines

I will add them in a separate patch in next version.

Okay.

Quoted 5 lines

I am still working on the documentation, which might be Documentation/media/uapi/v4l/pixfmt-meta-mtk-fd-40.rst. Refering the other pixfmt-*.rst files, I will try to provide the flat-table of the metadata with the structure of the mtk_fd_hw_result.

Sounds good to me.

Quoted 2 lines

I am confusing that should I remain the name with -40 in the tail of rst file?

The header and documentation file names should match the driver name. I just noticed there is some inconsistency in the naming, though. The driver seems to be located under drivers/media/platform/mtk-isp/fd, but the driver name in the platform driver struct and as reported by VIDIOC_QUERYCAP seems to be "mtk-fd-4.0".

Since we have many mtk-* drivers in the tree currently, I think it might make sense to consolidate them under drivers/media/platform/mediatek, similarly to drivers/media/platform/qcom or /rockchip. But it could be done later, as a follow-up.

My suggestion would be to place the driver under drivers/media/platform/mtk-fd-40 and also rename the related Kconfig symbol to include the _40 suffix.

What do you think?

Quoted 4 lines

Since I think the layout of metadata might be different in the future mtk fd drivers. Maybe they create a new one or should they update the rst file when they are upstreaming?

I think we'll decide what to do when that happens. Depending on whether there is more face detection hardware available, we might be able to migrate to standard controls and metadata buffer formats at that time.

Quoted 2 lines

Other comments are almost fixed, I will inform you by this mail thread as soon as I finish the documentation of fd metadata format.

Okay, thanks.

Best regards,
Tomasz

#27 lore raw

Hi Tomasz,

Quoted 79 lines · Tomasz Figa

Hi Jerry,

Quoted 44 lines · Jerry-ch Chen

Hi Tomasz,

Quoted 26 lines · Tomasz Figa

Hi Jerry,

Quoted 4 lines · Tomasz Figa

Hi Jerry,

Quoted 1 line · Jerry-ch Chen
Quoted 9 lines
Quoted 5 lines

Hi Laurent, Tomasz, Matthias,

gentle ping for this patch set,
If no new comments, I would like to send a newer version.

Sorry, I still haven't had a chance to look at the series, so feel free to send a new version and I will take a look at the new one.

Finally found some time to review the series. Again sorry for the delay and thanks for your patience.

Some general comments:
1) The metadata format FourCC should be added in a separate patch, together with documentation for it.
2) Control IDs, structs used by the userspace, etc. should be defined in a header under include/uapi/linux.

Quoted 4 lines

Please also check my replies to particular patches for further comments.

Best regards,
Tomasz

Appreciate for your reply,

So far, I've locally created an uapi header:
include/uapi/linux/mtk_fd_40.h
which provides some values, control ids, and the definitions of structures that would be needed by user of mtk_fd_40 driver. In addition, I also provide a MACRO as example in comments that can extract the struct member with bit length and offset definitions(eliminate the bit-fields).

Quoted 2 lines

Also, I would like to rename struct fd_user_output with struct mtk_fd_hw_result. I worry fd_user_output would be a confusing name.

The change sounds good to me.

Quoted 2 lines

I will add them in a separate patch in next version.

Okay.

Quoted 5 lines

I am still working on the documentation, which might be Documentation/media/uapi/v4l/pixfmt-meta-mtk-fd-40.rst. Refering the other pixfmt-*.rst files, I will try to provide the flat-table of the metadata with the structure of the mtk_fd_hw_result.

Sounds good to me.

Quoted 2 lines

I am confusing that should I remain the name with -40 in the tail of rst file?

The header and documentation file names should match the driver name. I just noticed there is some inconsistency in the naming, though. The driver seems to be located under drivers/media/platform/mtk-isp/fd, but the driver name in the platform driver struct and as reported by VIDIOC_QUERYCAP seems to be "mtk-fd-4.0".

Since we have many mtk-* drivers in the tree currently, I think it might make sense to consolidate them under drivers/media/platform/mediatek, similarly to drivers/media/platform/qcom or /rockchip. But it could be done later, as a follow-up.

Quoted 6 lines

My suggestion would be to place the driver under drivers/media/platform/mtk-fd-40 and also rename the related Kconfig symbol to include the _40 suffix.

What do you think?

I Appreciate your comments,
Sorry for the late reply.

Would it be possible for me to replace the driver as drivers/media/platform/mtk_fd/mtk-fd-40?(Just like mtk-isp/isp_50)

Quoted 9 lines
Quoted 4 lines

Since I think the layout of metadata might be different in the future mtk fd drivers. Maybe they create a new one or should they update the rst file when they are upstreaming?

I think we'll decide what to do when that happens. Depending on whether there is more face detection hardware available, we might be able to migrate to standard controls and metadata buffer formats at that time.

Ok.

Quoted 7 lines
Quoted 2 lines

Other comments are almost fixed, I will inform you by this mail thread as soon as I finish the documentation of fd metadata format.

Okay, thanks.

Best regards,
Tomasz

Thanks and Best Regards,
Jerry

#28 lore raw
Quoted 87 lines · Jerry-ch Chen

Hi Tomasz,

Quoted 83 lines · Tomasz Figa

Hi Jerry,

Quoted 44 lines · Jerry-ch Chen

Hi Tomasz,

Quoted 26 lines · Tomasz Figa

Hi Jerry,

Quoted 4 lines · Tomasz Figa

Hi Jerry,

Quoted 1 line · Jerry-ch Chen
Quoted 9 lines
Quoted 5 lines

Hi Laurent, Tomasz, Matthias,

gentle ping for this patch set,
If no new comments, I would like to send a newer version.

Sorry, I still haven't had a chance to look at the series, so feel free to send a new version and I will take a look at the new one.

Finally found some time to review the series. Again sorry for the delay and thanks for your patience.

Some general comments:
1) The metadata format FourCC should be added in a separate patch, together with documentation for it.
2) Control IDs, structs used by the userspace, etc. should be defined in a header under include/uapi/linux.

Quoted 4 lines

Please also check my replies to particular patches for further comments.

Best regards,
Tomasz

Appreciate for your reply,

So far, I've locally created an uapi header:
include/uapi/linux/mtk_fd_40.h
which provides some values, control ids, and the definitions of structures that would be needed by user of mtk_fd_40 driver. In addition, I also provide a MACRO as example in comments that can extract the struct member with bit length and offset definitions(eliminate the bit-fields).

Quoted 2 lines

Also, I would like to rename struct fd_user_output with struct mtk_fd_hw_result. I worry fd_user_output would be a confusing name.

The change sounds good to me.

Quoted 2 lines

I will add them in a separate patch in next version.

Okay.

Quoted 5 lines

I am still working on the documentation, which might be Documentation/media/uapi/v4l/pixfmt-meta-mtk-fd-40.rst. Refering the other pixfmt-*.rst files, I will try to provide the flat-table of the metadata with the structure of the mtk_fd_hw_result.

Sounds good to me.

Quoted 2 lines

I am confusing that should I remain the name with -40 in the tail of rst file?

The header and documentation file names should match the driver name. I just noticed there is some inconsistency in the naming, though. The driver seems to be located under drivers/media/platform/mtk-isp/fd, but the driver name in the platform driver struct and as reported by VIDIOC_QUERYCAP seems to be "mtk-fd-4.0".

Since we have many mtk-* drivers in the tree currently, I think it might make sense to consolidate them under drivers/media/platform/mediatek, similarly to drivers/media/platform/qcom or /rockchip. But it could be done later, as a follow-up.

My suggestion would be to place the driver under drivers/media/platform/mtk-fd-40 and also rename the related Kconfig symbol to include the _40 suffix.

Quoted 8 lines
Quoted 2 lines

What do you think?

I Appreciate your comments,
Sorry for the late reply.

Would it be possible for me to replace the driver as drivers/media/platform/mtk_fd/mtk-fd-40?(Just like mtk-isp/isp_50)

I'm not a big fan of duplicating "mtk fd" in the path. How about just making it drivers/media/platform/mtk-fd-40?

Best regards,
Tomasz

_______________________________________________
Linux-mediatek mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-mediatek

#29 lore raw

Hi Tomasz,

Quoted 92 lines · Tomasz Figa
Quoted 90 lines · Jerry-ch Chen

Hi Tomasz,

Quoted 85 lines · Tomasz Figa

Hi Jerry,

Quoted 44 lines · Jerry-ch Chen

Hi Tomasz,

Quoted 26 lines · Tomasz Figa

Hi Jerry,

Quoted 4 lines · Tomasz Figa

Hi Jerry,

Quoted 1 line · Jerry-ch Chen
Quoted 9 lines
Quoted 5 lines

Hi Laurent, Tomasz, Matthias,

gentle ping for this patch set,
If no new comments, I would like to send a newer version.

Sorry, I still haven't had a chance to look at the series, so feel free to send a new version and I will take a look at the new one.

Finally found some time to review the series. Again sorry for the delay and thanks for your patience.

Some general comments:
1) The metadata format FourCC should be added in a separate patch, together with documentation for it.
2) Control IDs, structs used by the userspace, etc. should be defined in a header under include/uapi/linux.

Quoted 4 lines

Please also check my replies to particular patches for further comments.

Best regards,
Tomasz

Appreciate for your reply,

So far, I've locally created an uapi header:
include/uapi/linux/mtk_fd_40.h
which provides some values, control ids, and the definitions of structures that would be needed by user of mtk_fd_40 driver. In addition, I also provide a MACRO as example in comments that can extract the struct member with bit length and offset definitions(eliminate the bit-fields).

Quoted 2 lines

Also, I would like to rename struct fd_user_output with struct mtk_fd_hw_result. I worry fd_user_output would be a confusing name.

The change sounds good to me.

Quoted 2 lines

I will add them in a separate patch in next version.

Okay.

Quoted 5 lines

I am still working on the documentation, which might be Documentation/media/uapi/v4l/pixfmt-meta-mtk-fd-40.rst. Refering the other pixfmt-*.rst files, I will try to provide the flat-table of the metadata with the structure of the mtk_fd_hw_result.

Sounds good to me.

Quoted 2 lines

I am confusing that should I remain the name with -40 in the tail of rst file?

The header and documentation file names should match the driver name. I just noticed there is some inconsistency in the naming, though. The driver seems to be located under drivers/media/platform/mtk-isp/fd, but the driver name in the platform driver struct and as reported by VIDIOC_QUERYCAP seems to be "mtk-fd-4.0".

Since we have many mtk-* drivers in the tree currently, I think it might make sense to consolidate them under drivers/media/platform/mediatek, similarly to drivers/media/platform/qcom or /rockchip. But it could be done later, as a follow-up.

My suggestion would be to place the driver under drivers/media/platform/mtk-fd-40 and also rename the related Kconfig symbol to include the _40 suffix.

What do you think?

Quoted 9 lines
Quoted 5 lines

I Appreciate your comments,
Sorry for the late reply.

Would it be possible for me to replace the driver as drivers/media/platform/mtk_fd/mtk-fd-40?(Just like mtk-isp/isp_50)

I'm not a big fan of duplicating "mtk fd" in the path. How about just making it drivers/media/platform/mtk-fd-40?

Ok, I will make it drivers/media/platform/mtk-fd-40,
and also rename the related Kconfig symbol to include the _40 suffix.

Thanks and Best Regards,
Jerry.

Quoted 2 lines

Best regards,
Tomasz

#30 lore raw

Hi Tomasz,

Quoted 105 lines · Jerry-ch Chen

Hi Tomasz,

Quoted 92 lines · Tomasz Figa
Quoted 90 lines · Jerry-ch Chen

Hi Tomasz,

Quoted 85 lines · Tomasz Figa

Hi Jerry,

Quoted 44 lines · Jerry-ch Chen

Hi Tomasz,

Quoted 26 lines · Tomasz Figa

Hi Jerry,

Quoted 4 lines · Tomasz Figa

Hi Jerry,

Quoted 1 line · Jerry-ch Chen
Quoted 9 lines
Quoted 5 lines

Hi Laurent, Tomasz, Matthias,

gentle ping for this patch set,
If no new comments, I would like to send a newer version.

Sorry, I still haven't had a chance to look at the series, so feel free to send a new version and I will take a look at the new one.

Finally found some time to review the series. Again sorry for the delay and thanks for your patience.

Some general comments:
1) The metadata format FourCC should be added in a separate patch, together with documentation for it.
2) Control IDs, structs used by the userspace, etc. should be defined in a header under include/uapi/linux.

Quoted 4 lines

Please also check my replies to particular patches for further comments.

Best regards,
Tomasz

Appreciate for your reply,

So far, I've locally created an uapi header:
include/uapi/linux/mtk_fd_40.h
which provides some values, control ids, and the definitions of structures that would be needed by user of mtk_fd_40 driver. In addition, I also provide a MACRO as example in comments that can extract the struct member with bit length and offset definitions(eliminate the bit-fields).

Quoted 2 lines

Also, I would like to rename struct fd_user_output with struct mtk_fd_hw_result. I worry fd_user_output would be a confusing name.

The change sounds good to me.

Quoted 2 lines

I will add them in a separate patch in next version.

Okay.

Quoted 5 lines

I am still working on the documentation, which might be Documentation/media/uapi/v4l/pixfmt-meta-mtk-fd-40.rst. Refering the other pixfmt-*.rst files, I will try to provide the flat-table of the metadata with the structure of the mtk_fd_hw_result.

Sounds good to me.

Quoted 2 lines

I am confusing that should I remain the name with -40 in the tail of rst file?

The header and documentation file names should match the driver name. I just noticed there is some inconsistency in the naming, though. The driver seems to be located under drivers/media/platform/mtk-isp/fd, but the driver name in the platform driver struct and as reported by VIDIOC_QUERYCAP seems to be "mtk-fd-4.0".

Since we have many mtk-* drivers in the tree currently, I think it might make sense to consolidate them under drivers/media/platform/mediatek, similarly to drivers/media/platform/qcom or /rockchip. But it could be done later, as a follow-up.

My suggestion would be to place the driver under drivers/media/platform/mtk-fd-40 and also rename the related Kconfig symbol to include the _40 suffix.

What do you think?

Quoted 9 lines
Quoted 5 lines

I Appreciate your comments,
Sorry for the late reply.

Would it be possible for me to replace the driver as drivers/media/platform/mtk_fd/mtk-fd-40?(Just like mtk-isp/isp_50)

I'm not a big fan of duplicating "mtk fd" in the path. How about just making it drivers/media/platform/mtk-fd-40?

Quoted 9 lines

Ok, I will make it drivers/media/platform/mtk-fd-40,
and also rename the related Kconfig symbol to include the _40 suffix.

Thanks and Best Regards,
Jerry.

Quoted 2 lines

Best regards,
Tomasz

I've finish the document of FD driver, describing the structure of the mtk_fd_hw_result. Could I send the new version of the driver? would the folder path replacement must be included in the new version?

Thanks and Best Regards,
Jerry
_______________________________________________
Linux-mediatek mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-mediatek

#31 lore raw
Quoted 115 lines · Jerry-ch Chen

Hi Tomasz,

Quoted 111 lines · Jerry-ch Chen

Hi Tomasz,

Quoted 92 lines · Tomasz Figa
Quoted 90 lines · Jerry-ch Chen

Hi Tomasz,

Quoted 85 lines · Tomasz Figa

Hi Jerry,

Quoted 44 lines · Jerry-ch Chen

Hi Tomasz,

Quoted 26 lines · Tomasz Figa

Hi Jerry,

Quoted 4 lines · Tomasz Figa

Hi Jerry,

Quoted 1 line · Jerry-ch Chen
Quoted 9 lines
Quoted 5 lines

Hi Laurent, Tomasz, Matthias,

gentle ping for this patch set,
If no new comments, I would like to send a newer version.

Sorry, I still haven't had a chance to look at the series, so feel free to send a new version and I will take a look at the new one.

Finally found some time to review the series. Again sorry for the delay and thanks for your patience.

Some general comments:
1) The metadata format FourCC should be added in a separate patch, together with documentation for it.
2) Control IDs, structs used by the userspace, etc. should be defined in a header under include/uapi/linux.

Quoted 4 lines

Please also check my replies to particular patches for further comments.

Best regards,
Tomasz

Appreciate for your reply,

So far, I've locally created an uapi header:
include/uapi/linux/mtk_fd_40.h
which provides some values, control ids, and the definitions of structures that would be needed by user of mtk_fd_40 driver. In addition, I also provide a MACRO as example in comments that can extract the struct member with bit length and offset definitions(eliminate the bit-fields).

Quoted 2 lines

Also, I would like to rename struct fd_user_output with struct mtk_fd_hw_result. I worry fd_user_output would be a confusing name.

The change sounds good to me.

Quoted 2 lines

I will add them in a separate patch in next version.

Okay.

Quoted 5 lines

I am still working on the documentation, which might be Documentation/media/uapi/v4l/pixfmt-meta-mtk-fd-40.rst. Refering the other pixfmt-*.rst files, I will try to provide the flat-table of the metadata with the structure of the mtk_fd_hw_result.

Sounds good to me.

Quoted 2 lines

I am confusing that should I remain the name with -40 in the tail of rst file?

The header and documentation file names should match the driver name. I just noticed there is some inconsistency in the naming, though. The driver seems to be located under drivers/media/platform/mtk-isp/fd, but the driver name in the platform driver struct and as reported by VIDIOC_QUERYCAP seems to be "mtk-fd-4.0".

Since we have many mtk-* drivers in the tree currently, I think it might make sense to consolidate them under drivers/media/platform/mediatek, similarly to drivers/media/platform/qcom or /rockchip. But it could be done later, as a follow-up.

My suggestion would be to place the driver under drivers/media/platform/mtk-fd-40 and also rename the related Kconfig symbol to include the _40 suffix.

What do you think?

Quoted 9 lines
Quoted 5 lines

I Appreciate your comments,
Sorry for the late reply.

Would it be possible for me to replace the driver as drivers/media/platform/mtk_fd/mtk-fd-40?(Just like mtk-isp/isp_50)

I'm not a big fan of duplicating "mtk fd" in the path. How about just making it drivers/media/platform/mtk-fd-40?

Ok, I will make it drivers/media/platform/mtk-fd-40,
and also rename the related Kconfig symbol to include the _40 suffix.

Thanks and Best Regards,
Jerry.

Quoted 8 lines
Quoted 3 lines
Quoted 2 lines

Best regards,
Tomasz

I've finish the document of FD driver, describing the structure of the mtk_fd_hw_result. Could I send the new version of the driver? would the folder path replacement must be included in the new version?

Thanks Jerry and Happy New Year.

Since the driver should be in a good shape after addressing the last comments, perhaps it would make sense to update the path as well, so that it can be merged if no other issues are found?

Best regards,
Tomasz