Skip to contentSkip to conversation

media: imx: staging cleanup and FIM export to v4l2-core

PATCH Superseded Jiaqi Shi 2 participants 7 messages Nov 21

Subsystems (4)

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Jiaqi Shi

author
#0 lore raw

cover media: imx: staging cleanup and FIM export to v4l2-core

This series continues the i.MX staging media driver cleanup by removing legacy video node handling and exporting the Frame Interval Monitor to v4l2-core.

Patch 1 removes control inheritance and source-dependent format handling from capture video nodes, as both are deprecated in favor of the subdev-based userspace API.

Patch 2 moves the generic FIM implementation to v4l2-core so other drivers can reuse it, while keeping the i.MX-specific input capture controls in the i.MX adapter. The error calculation uses signed arithmetic to avoid underflow when the measured interval is smaller than the nominal interval.

Patch 3 allows the FIM input capture channel control to be written, which was previously rejected with -EINVAL.

Compile-tested with allmodconfig; checkpatch.pl --strict is clean. No runtime testing due to lack of hardware.

Jiaqi Shi (3):
media: imx: remove legacy video node control and format handling
media: v4l2-core: add frame interval monitor helper
media: imx: allow setting FIM input capture channel

 Documentation/admin-guide/media/imx.rst       |  10 +-
 drivers/media/v4l2-core/Kconfig               |   4 +
 drivers/media/v4l2-core/Makefile              |   1 +
 drivers/media/v4l2-core/v4l2-fim.c            | 324 ++++++++++++++++
 drivers/staging/media/imx/Kconfig             |   1 +
 drivers/staging/media/imx/TODO                |  13 -
 drivers/staging/media/imx/imx-ic-prpencvf.c   |   5 +-
 drivers/staging/media/imx/imx-media-capture.c | 192 +---------
 drivers/staging/media/imx/imx-media-csi.c     |   5 +-
 .../staging/media/imx/imx-media-dev-common.c  | 113 +-----
 drivers/staging/media/imx/imx-media-fim.c     | 355 +++---------------
 drivers/staging/media/imx/imx-media.h         |   7 +-
 include/media/v4l2-fim.h                      |  40 ++
 13 files changed, 444 insertions(+), 626 deletions(-)
 create mode 100644 drivers/media/v4l2-core/v4l2-fim.c
 delete mode 100644 drivers/staging/media/imx/TODO
 create mode 100644 include/media/v4l2-fim.h
Signature · 1 line
2.53.0
author
#1 lore raw

PATCH 1/3 media: imx: remove legacy video node control and format handling

Remove control inheritance from pipeline subdevices to video nodes. Userspace should access controls through subdev nodes instead.

Use source-independent format ioctls for all capture nodes and initialize them to the existing UYVY 640x480 default instead of the source subdev active format. Keep stream-time format validation.

Update the usage documentation and remove the completed TODO entries.

---
 Documentation/admin-guide/media/imx.rst       |  10 +-
 drivers/staging/media/imx/TODO                |  10 -
 drivers/staging/media/imx/imx-ic-prpencvf.c   |   5 +-
 drivers/staging/media/imx/imx-media-capture.c | 192 ++----------------
 drivers/staging/media/imx/imx-media-csi.c     |   5 +-
 .../staging/media/imx/imx-media-dev-common.c  | 113 +----------
 drivers/staging/media/imx/imx-media.h         |   7 +-
 7 files changed, 21 insertions(+), 321 deletions(-)
Documentation/admin-guide/media/imx.rst +3 -7
index bb68100d8acb..b1c5fecfaef7 100644
@@ -394,13 +394,9 @@ scaling up to 1024x1024, CSC, flip, and rotation:
 Usage Notes
 -----------
 
-To aid in configuration and for backward compatibility with V4L2
-applications that access controls only from video device nodes, the
-capture device interfaces inherit controls from the active entities
-in the current pipeline, so controls can be accessed either directly
-from the subdev or from the active capture device interface. For
-example, the FIM controls are available either from the ipuX_csiY
-subdevs or from the active capture device.
+Controls for pipeline entities are accessed through their subdev device
+nodes. For example, the FIM controls are available from the ipuX_csiY
+subdevs.
 
 The following are specific usage notes for the Sabre* reference
 boards:
drivers/staging/media/imx/TODO +0 -10
index 11c9e10d34ae..43cd030d4bbd 100644
@@ -1,13 +1,3 @@
 
 - The Frame Interval Monitor could be exported to v4l2-core for
   general use.
-
-- This media driver supports inheriting V4L2 controls to the
-  video capture devices, from the subdevices in the capture device's
-  pipeline. The controls for each capture device are updated in the
-  link_notify callback when the pipeline is modified. This feature should be
-  removed, userspace should use the subdev-based userspace API instead.
-
-- Similarly to the legacy control handling, legacy format handling where
-  formats on the video nodes are influenced by the active format of the
-  connected subdev should be removed.
drivers/staging/media/imx/imx-ic-prpencvf.c +1 -4
index 3430ddeac356..b3f2177e1a5d 100644
@@ -1286,7 +1286,7 @@ static int prp_registered(struct v4l2_subdev *sd)
 
 	ret = imx_media_capture_device_register(priv->vdev, 0);
 	if (ret)
-		goto remove_vdev;
+		return ret;
 
 	ret = prp_init_controls(priv);
 	if (ret)
@@ -1296,8 +1296,6 @@ static int prp_registered(struct v4l2_subdev *sd)
 
 unreg_vdev:
 	imx_media_capture_device_unregister(priv->vdev);
-remove_vdev:
-	imx_media_capture_device_remove(priv->vdev);
 	return ret;
 }
 
@@ -1306,7 +1304,6 @@ static void prp_unregistered(struct v4l2_subdev *sd)
 	struct prp_priv *priv = sd_to_priv(sd);
 
 	imx_media_capture_device_unregister(priv->vdev);
-	imx_media_capture_device_remove(priv->vdev);
 
 	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
 }
drivers/staging/media/imx/imx-media-capture.c +13 -179

257 lines not on this page · raw message

drivers/staging/media/imx/imx-media-csi.c +1 -4
index 43c637b333f2..3a285fd10ac9 100644
@@ -1823,12 +1823,10 @@ static int csi_registered(struct v4l2_subdev *sd)
 
 	ret = imx_media_capture_device_register(priv->vdev, 0);
 	if (ret)
-		goto remove_vdev;
+		goto free_fim;
 
 	return 0;
 
-remove_vdev:
-	imx_media_capture_device_remove(priv->vdev);
 free_fim:
 	if (priv->fim)
 		imx_media_fim_free(priv->fim);
@@ -1842,7 +1840,6 @@ static void csi_unregistered(struct v4l2_subdev *sd)
 	struct csi_priv *priv = v4l2_get_subdevdata(sd);
 
 	imx_media_capture_device_unregister(priv->vdev);
-	imx_media_capture_device_remove(priv->vdev);
 
 	if (priv->fim)
 		imx_media_fim_free(priv->fim);
drivers/staging/media/imx/imx-media-dev-common.c +1 -112

131 lines not on this page · raw message

drivers/staging/media/imx/imx-media.h +2 -5
index 537558a7bece..832c9d292353 100644
@@ -125,10 +125,8 @@ static inline struct imx_media_buffer *to_imx_media_vb(struct vb2_buffer *vb)
 }
 
 /*
- * to support control inheritance to video devices, this
- * retrieves a pad's list_head of video devices that can
- * be reached from the pad. Note that only the lists in
- * source pads get populated, sink pads have empty lists.
+ * Retrieve a pad's list of video devices reachable from the pad for
+ * subdev event forwarding. Only source pad lists are populated.
  */
 static inline struct list_head *
 to_pad_vdev_list(struct v4l2_subdev *sd, int pad_index)
@@ -271,7 +269,6 @@ int imx_media_ic_unregister(struct v4l2_subdev *sd);
 struct imx_media_video_dev *
 imx_media_capture_device_init(struct device *dev, struct v4l2_subdev *src_sd,
 			      int pad, bool legacy_api);
-void imx_media_capture_device_remove(struct imx_media_video_dev *vdev);
 int imx_media_capture_device_register(struct imx_media_video_dev *vdev,
 				      u32 link_flags);
 void imx_media_capture_device_unregister(struct imx_media_video_dev *vdev);
Signature · 1 line
2.53.0
#2 lore raw

Re: [PATCH 1/3] media: imx: remove legacy video node control and format handling

Thank you for your contribution! Sashiko AI review found 5 potential issues to consider: - [High] media: imx: unresolved default colorimetry and field in capture_init_format() - [High] media: imx…

Thank you for your contribution! Sashiko AI review found 5 potential issues to consider:
- [High] media: imx: unresolved default colorimetry and field in capture_init_format()
- [High] media: imx: insufficient pixel format validation in capture_validate_fmt()
- [High] media: imx: unclamped dimensions in __capture_try_fmt() cause divide-by-zero
- [Medium] media: imx: callers of imx_media_capture_device_init() still set legacy_api
- [Low] media: imx: outdated v4l2-ctl format examples in Documentation/admin-guide
--

commit 91db31570bbb5c6e57f4f438a93a25e73be6926a
Author: Jiaqi Shi <[email protected]>

media: imx: remove legacy video node control and format handling

This commit removes V4L2 control inheritance from pipeline subdevices to capture video nodes and switches capture video nodes to source-independent format initialization and format ioctls.

Quoted 8 lines
Documentation/admin-guide/media/imx.rst +0 -0
index bb68100d8acba..b1c5fecfaef74 100644
@@ -394,13 +394,9 @@ scaling up to 1024x1024, CSC, flip, and rotation:
 Usage Notes
 -----------
 

Documentation/admin-guide/media/imx.rst scaling up to 1024x1024, CSC, flip, and rotation: · line 399

Quoted 10 lines
Documentation/admin-guide/media/imx.rst +3 -7
-To aid in configuration and for backward compatibility with V4L2
-applications that access controls only from video device nodes, the
-capture device interfaces inherit controls from the active entities
-in the current pipeline, so controls can be accessed either directly
-from the subdev or from the active capture device interface. For
-example, the FIM controls are available either from the ipuX_csiY
-subdevs or from the active capture device.
+Controls for pipeline entities are accessed through their subdev device
+nodes. For example, the FIM controls are available from the ipuX_csiY
+subdevs.

[Severity: Low]
Should the v4l2-ctl --set-fmt-video examples later in this document also be updated to specify width and height when the upstream subdev pad is not 640x480?

Because capture nodes now initialize to 640x480 and VIDIOC_S_FMT no longer inherits dimensions from the source subdev, the 720x480, 720x576, and 800x600 pipeline examples that only pass field= or pixelformat=RGB3 leave the capture node at 640x480 and fail in capture_validate_fmt() with -EPIPE at stream start.

[ ... ]

Quoted 4 lines
drivers/staging/media/imx/imx-media-capture.c +0 -0
index 9216a221b72f4..93d05cd967980 100644

[ ... ]

drivers/staging/media/imx/imx-media-capture.c static const struct v4l2_ioctl_ops capture_legacy_ioctl_ops = { · line 438

Quoted 11 lines
drivers/staging/media/imx/imx-media-capture.c +3 -3
@@ -566,10 +432,10 @@ static const struct v4l2_ioctl_ops capture_legacy_ioctl_ops = {
 	.vidioc_enum_framesizes		= capture_legacy_enum_framesizes,
 	.vidioc_enum_frameintervals	= capture_legacy_enum_frameintervals,
 
-	.vidioc_enum_fmt_vid_cap	= capture_legacy_enum_fmt_vid_cap,
+	.vidioc_enum_fmt_vid_cap	= capture_enum_fmt_vid_cap,
 	.vidioc_g_fmt_vid_cap		= capture_g_fmt_vid_cap,
-	.vidioc_try_fmt_vid_cap		= capture_legacy_try_fmt_vid_cap,
-	.vidioc_s_fmt_vid_cap		= capture_legacy_s_fmt_vid_cap,
+	.vidioc_try_fmt_vid_cap		= capture_try_fmt_vid_cap,
+	.vidioc_s_fmt_vid_cap		= capture_s_fmt_vid_cap,

[Severity: High]
Does capture_validate_fmt() allow incompatible pixel formats on the video node now that capture_s_fmt_vid_cap() accepts any PIXFMT_SEL_ANY format?

Previously, capture_legacy_s_fmt_vid_cap() called capture_find_format() with both fmt_src->format.code and pixfmt->pixelformat, which rejected Bayer formats on IPU internal links and required an exact format match on non-IPU passthrough links. Now capture_validate_fmt() is the only check, and it only compares the two-value colorspace enum:

drivers/staging/media/imx/imx-media-capture.c:capture_validate_fmt() {
    ...
	cc = capture_find_format(fmt_src.format.code, 0);
	if (!cc || priv->vdev.cc->cs != cc->cs)
		return -EPIPE;
    ...
}

Because all RGB, Bayer, and greyscale formats share IPUV3_COLORSPACE_RGB and all YUV formats share IPUV3_COLORSPACE_YUV, mismatched formats pass validation. On a CSI passthrough link, configuring an 8-bpp or 12-bpp format on the video node while the source subdev outputs a 16-bpp format with the same colorspace programs 16-bpp IDMAC DMA transfers into buffers sized for 8 or 12 bpp, overflowing the DMA buffer. On an IPU internal RGB link, V4L2_PIX_FMT_Y10 or V4L2_PIX_FMT_Y12 reaches the default WARN_ON(1) in ipu_cpmem_set_image(), while 8-bpp Bayer or greyscale formats cause ipu_cpmem_set_fmt() to fail with its return value ignored, leaving the CPMEM BPP field at 0 (32 bpp) for 8-bpp buffers.

Passing priv->vdev.fmt.pixelformat instead of 0 to capture_find_format() in capture_validate_fmt() would restore the format check.

[Severity: High]
Can VIDIOC_S_FMT followed by VIDIOC_REQBUFS trigger a divide-by-zero in capture_queue_setup()?

capture_legacy_s_fmt_vid_cap() previously took width and height from the source subdev, where dimensions were already clamped to non-zero hardware bounds. capture_s_fmt_vid_cap() instead passes the user's unclamped width and height through __capture_try_fmt() to imx_media_mbus_fmt_to_pix_fmt(), so width = 0 or height = 0 sets priv->vdev.fmt.sizeimage to 0. A subsequent VIDIOC_REQBUFS or VIDIOC_CREATE_BUFS then divides by pix->sizeimage:

drivers/staging/media/imx/imx-media-capture.c:capture_queue_setup() {
    ...
	count = min_t(__u32, VID_MEM_LIMIT / pix->sizeimage, count);
    ...
}

[ ... ]

Quoted 34 lines
drivers/staging/media/imx/imx-media-capture.c +6 -21
@@ -869,36 +735,21 @@ void imx_media_capture_device_error(struct imx_media_video_dev *vdev)
 }
 EXPORT_SYMBOL_GPL(imx_media_capture_device_error);
 
-static int capture_init_format(struct capture_priv *priv)
+static void capture_init_format(struct capture_priv *priv)
 {
-	struct v4l2_subdev_format fmt_src = {
-		.pad = priv->src_sd_pad,
-		.which = V4L2_SUBDEV_FORMAT_ACTIVE,
-	};
 	struct imx_media_video_dev *vdev = &priv->vdev;
-	int ret;
-
-	if (priv->legacy_api) {
-		ret = v4l2_subdev_call(priv->src_sd, pad, get_fmt, NULL,
-				       &fmt_src);
-		if (ret) {
-			dev_err(priv->dev, "failed to get source format\n");
-			return ret;
-		}
-	} else {
-		fmt_src.format.code = MEDIA_BUS_FMT_UYVY8_2X8;
-		fmt_src.format.width = IMX_MEDIA_DEF_PIX_WIDTH;
-		fmt_src.format.height = IMX_MEDIA_DEF_PIX_HEIGHT;
-	}
+	struct v4l2_mbus_framefmt fmt_src = {
+		.code = MEDIA_BUS_FMT_UYVY8_2X8,
+		.width = IMX_MEDIA_DEF_PIX_WIDTH,
+		.height = IMX_MEDIA_DEF_PIX_HEIGHT,
+	};
 
-	imx_media_mbus_fmt_to_pix_fmt(&vdev->fmt, &fmt_src.format, NULL);
-	vdev->compose.width = fmt_src.format.width;

drivers/staging/media/imx/imx-media-capture.c void imx_media_capture_device_error(struct imx_media_video_dev *vdev) · line 753

Quoted 10 lines
drivers/staging/media/imx/imx-media-capture.c +3 -3
-	vdev->compose.height = fmt_src.format.height;
+	imx_media_mbus_fmt_to_pix_fmt(&vdev->fmt, &fmt_src, NULL);
+	vdev->compose.width = fmt_src.width;
+	vdev->compose.height = fmt_src.height;
 
 	vdev->cc = imx_media_find_pixel_format(vdev->fmt.pixelformat,
 					       PIXFMT_SEL_ANY);
-
-	return 0;
 }

[Severity: High]
Does leaving fmt_src.field and the colorimetry members zero-initialized in capture_init_format() and __capture_try_fmt() break IC color space conversion at stream start?

Previously, vdev->fmt was initialized and updated from the source subdev's active format, which imx_media_init_mbus_fmt() and imx_media_try_colorimetry() populated with V4L2_FIELD_NONE and resolved colorimetry values (such as V4L2_YCBCR_ENC_601 and V4L2_QUANTIZATION_LIM_RANGE). Now fmt_src leaves field as V4L2_FIELD_ANY
(0) and ycbcr_enc and quantization as DEFAULT (0), and __capture_try_fmt() also passes zeroed values through without resolving them.

When streaming starts on an IC capture node (ipuX_ic_prpenc or ipuX_ic_prpvf), prp_setup_norotation() and prp_setup_rotation() pass vdev->fmt.ycbcr_enc and vdev->fmt.quantization to ipu_ic_calc_csc() -> calc_csc_coeffs(), which rejects V4L2_YCBCR_ENC_DEFAULT with -ENOTSUPP whenever YUV-to-RGB or RGB-to-YUV conversion is needed:

drivers/gpu/ipu-v3/ipu-ic-csc.c:calc_csc_coeffs() {
    ...
	switch (csc->out_cs.enc) {
	case V4L2_YCBCR_ENC_601:
		params_tbl = (csc->in_cs.cs == IPUV3_COLORSPACE_YUV) ?
			yuv2rgb_601 : rgb2yuv_601;
		break;
	case V4L2_YCBCR_ENC_709:
		params_tbl = (csc->in_cs.cs == IPUV3_COLORSPACE_YUV) ?
			yuv2rgb_709 : rgb2yuv_709;
		break;
	default:
		return -ENOTSUPP;
	}
    ...
}

On default YUV-to-YUV IC captures, V4L2_QUANTIZATION_DEFAULT also maps to full range instead of limited range, programming the yuvl2yuvf range expansion CSC matrix instead of an identity pass-through.

[ ... ]

drivers/staging/media/imx/imx-media-capture.c imx_media_capture_device_init(struct device *dev, struct v4l2_subdev *src_sd, · line 977

Quoted 5 lines
drivers/staging/media/imx/imx-media-capture.c +0 -1
@@ -974,7 +823,6 @@ imx_media_capture_device_init(struct device *dev, struct v4l2_subdev *src_sd,
 	priv->src_sd = src_sd;
 	priv->src_sd_pad = pad;
 	priv->dev = dev;
-	priv->legacy_api = legacy_api;

[Severity: Medium]
Should prp_registered() and csi_registered() stop setting legacy_api = true (or should legacy_api be removed altogether)?

Both callers of imx_media_capture_device_init() still pass legacy_api = true, which leaves V4L2_CAP_IO_MC unset on every capture device, keeps capture_ioctl_ops unused, and keeps capture_legacy_enum_framesizes() active. Without V4L2_CAP_IO_MC, v4l_enum_fmt() in the V4L2 core zeroes f->mbus_code before calling capture_enum_fmt_vid_cap(), and on ipuX_ic_prpenc/prpvf capture nodes capture_legacy_enum_framesizes() passes non-IPU mbus codes to prp_enum_frame_size(), which rejects 22 of the 23 enumerated pixel formats with -EINVAL.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
author
#3 lore raw

PATCH 2/3 media: v4l2-core: add frame interval monitor helper

In reply to Jiaqi Shi

Move the generic frame interval monitoring implementation from i.MX to v4l2-core. The helper supports driver-selected control and event IDs and accepts EOF or external timestamps.

Keep the i.MX-specific input capture controls in the i.MX adapter, sharing the original control handler and preserving the existing UAPI and event semantics.

Select V4L2_FIM from VIDEO_IMX_MEDIA and remove the completed TODO.

---
 drivers/media/v4l2-core/Kconfig           |   4 +
 drivers/media/v4l2-core/Makefile          |   1 +
 drivers/media/v4l2-core/v4l2-fim.c        | 324 ++++++++++++++++++++
 drivers/staging/media/imx/Kconfig         |   1 +
 drivers/staging/media/imx/TODO            |   3 -
 drivers/staging/media/imx/imx-media-fim.c | 352 ++++------------------
 include/media/v4l2-fim.h                  |  40 +++
 7 files changed, 421 insertions(+), 304 deletions(-)
 create mode 100644 drivers/media/v4l2-core/v4l2-fim.c
 delete mode 100644 drivers/staging/media/imx/TODO
 create mode 100644 include/media/v4l2-fim.h
drivers/media/v4l2-core/Kconfig +4 -0
index d50ccac9733c..cff961b24c04 100644
@@ -40,6 +40,10 @@ config VIDEO_TUNER
 config V4L2_JPEG_HELPER
 	tristate
 
+# Used by drivers that need v4l2-fim.ko
+config V4L2_FIM
+	tristate
+
 # Used by drivers that need v4l2-h264.ko
 config V4L2_H264
 	tristate
drivers/media/v4l2-core/Makefile +1 -0
index 329f0eadce99..d5f6699a2669 100644
@@ -26,6 +26,7 @@ videodev-$(CONFIG_VIDEO_V4L2_I2C) += v4l2-i2c.o
 
 obj-$(CONFIG_V4L2_ASYNC) += v4l2-async.o
 obj-$(CONFIG_V4L2_CCI) += v4l2-cci.o
+obj-$(CONFIG_V4L2_FIM) += v4l2-fim.o
 obj-$(CONFIG_V4L2_FLASH_LED_CLASS) += v4l2-flash-led-class.o
 obj-$(CONFIG_V4L2_FWNODE) += v4l2-fwnode.o
 obj-$(CONFIG_V4L2_H264) += v4l2-h264.o
drivers/media/v4l2-core/v4l2-fim.c new file +324 -0

324 lines not on this page · raw message

drivers/staging/media/imx/Kconfig +1 -0
index 1cd48028b641..f559b9979039 100644
@@ -6,6 +6,7 @@ config VIDEO_IMX_MEDIA
 	depends on VIDEO_DEV
 	depends on IMX_IPUV3_CORE
 	select MEDIA_CONTROLLER
+	select V4L2_FIM
 	select V4L2_FWNODE
 	select V4L2_MEM2MEM_DEV
 	select VIDEOBUF2_DMA_CONTIG
drivers/staging/media/imx/TODO deleted +0 -3
deleted file mode 100644
index 43cd030d4bbd..000000000000
@@ -1,3 +0,0 @@
-
-- The Frame Interval Monitor could be exported to v4l2-core for
-  general use.
drivers/staging/media/imx/imx-media-fim.c +51 -301

464 lines not on this page · raw message

include/media/v4l2-fim.h new file +40 -0
new file mode 100644
index 000000000000..4596d0492676
@@ -0,0 +1,40 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * V4L2 frame interval monitor helpers
+ *
+ * Copyright (c) 2016 Mentor Graphics Inc.
+ */
+
+#ifndef _V4L2_FIM_H
+#define _V4L2_FIM_H
+
+#include <linux/ktime.h>
+#include <linux/types.h>
+
+struct v4l2_ctrl_handler;
+struct v4l2_fract;
+struct v4l2_subdev;
+
+struct v4l2_fim;
+
+#define V4L2_FIM_NUM_CONTROLS	5
+
+struct v4l2_fim_config {
+	u32 ctrl_base;
+	u32 event_type;
+	void *priv;
+	bool (*external_timestamp)(void *priv, bool curval);
+	void (*wait_first_timestamp)(void *priv);
+};
+
+struct v4l2_fim *v4l2_fim_init(struct v4l2_subdev *sd,
+			       struct v4l2_ctrl_handler *hdlr,
+			       const struct v4l2_fim_config *config);
+void v4l2_fim_reset(struct v4l2_fim *fim, bool curval);
+void v4l2_fim_set_stream(struct v4l2_fim *fim,
+			 const struct v4l2_fract *frame_interval, bool on);
+void v4l2_fim_eof_monitor(struct v4l2_fim *fim, ktime_t timestamp);
+void v4l2_fim_monitor_timestamp(struct v4l2_fim *fim, ktime_t timestamp);
+bool v4l2_fim_is_streaming(struct v4l2_fim *fim);
+
+#endif
Signature · 1 line
2.53.0
#4 lore raw

Re: [PATCH 2/3] media: v4l2-core: add frame interval monitor helper

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Medium] media: v4l2-fim: cross-cluster ctrl->val read in reset_fim() --

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] media: v4l2-fim: cross-cluster ctrl->val read in reset_fim() --

commit 4ee73622d4ea40f9f5507b8b2948de21bfddfccc
Author: Jiaqi Shi <[email protected]>

media: v4l2-core: add frame interval monitor helper

This commit extracts the frame interval monitoring helper from the i.MX staging driver into v4l2-core while keeping the input capture controls in imx-media-fim.c.

Quoted 6 lines
drivers/media/v4l2-core/v4l2-fim.c new file +0 -0
new file mode 100644
index 0000000000000..57d595196663b
@@ -0,0 +1,324 @@

[ ... ]

Quoted 15 lines
drivers/media/v4l2-core/v4l2-fim.c new file +15 -0
+static void reset_fim(struct v4l2_fim *fim, bool curval)
+{
+	struct v4l2_ctrl *en = fim->ctrl[FIM_CL_ENABLE];
+	struct v4l2_ctrl *num = fim->ctrl[FIM_CL_NUM];
+	struct v4l2_ctrl *skip = fim->ctrl[FIM_CL_NUM_SKIP];
+	struct v4l2_ctrl *tol_min = fim->ctrl[FIM_CL_TOLERANCE_MIN];
+	struct v4l2_ctrl *tol_max = fim->ctrl[FIM_CL_TOLERANCE_MAX];
+
+	if (fim->config.external_timestamp)
+		fim->external_timestamp =
+			fim->config.external_timestamp(fim->config.priv, curval);
+
+	if (curval) {
+		fim->enabled = en->cur.val;
+		fim->num_avg = num->cur.val;

drivers/media/v4l2-core/v4l2-fim.c line 88

Quoted 10 lines
drivers/media/v4l2-core/v4l2-fim.c new file +10 -0
+		fim->num_skip = skip->cur.val;
+		fim->tolerance_min = tol_min->cur.val;
+		fim->tolerance_max = tol_max->cur.val;
+	} else {
+		fim->enabled = en->val;
+		fim->num_avg = num->val;
+		fim->num_skip = skip->val;
+		fim->tolerance_min = tol_min->val;
+		fim->tolerance_max = tol_max->val;
+	}

[Severity: Medium]
Can reset_fim() and fim->config.external_timestamp() read uncommitted ctrl->val values across separate control clusters when curval is false?

v4l2_fim_init() clusters the five core FIM controls into fim->ctrl, while imx_media_fim_init() clusters the two input capture controls separately into fim->icap_ctrl. The V4L2 control framework only synchronizes ctrl->val with ctrl->cur.val for the cluster currently being tried or set, and leaves trial or rejected values in ctrl->val after VIDIOC_TRY_EXT_CTRLS or a failed s_ctrl.

If setting V4L2_CID_IMX_FIM_ICAP_EDGE while streaming fails with -EBUSY (or is tried via VIDIOC_TRY_EXT_CTRLS), icap_edge->val retains the uncommitted value. A subsequent update to any core FIM control calls fim_s_ctrl() -> reset_fim(fim, false) -> imx_fim_external_timestamp(..., false), which reads icap_edge->val and enables external_timestamp mode during active streaming, disabling EOF monitoring. Conversely, updating fim->icap_ctrl calls v4l2_fim_reset(fim->core, false), causing reset_fim() to read uncommitted ->val fields from the core fim->ctrl cluster.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
author
#5 lore raw

PATCH 3/3 media: imx: allow setting FIM input capture channel

In reply to Jiaqi Shi

fim_s_ctrl() rejected all controls except V4L2_CID_IMX_FIM_ICAP_EDGE with -EINVAL, which made V4L2_CID_IMX_FIM_ICAP_CHANNEL read-only in practice even though it is registered as a writable control.

Add an explicit case for ICAP_CHANNEL so it can be updated, and fix the wait_for_completion_timeout() continuation style while here.

---
 drivers/staging/media/imx/imx-media-fim.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
drivers/staging/media/imx/imx-media-fim.c +4 -3
index b6467e29ac28..300635a3b552 100644
@@ -58,9 +58,8 @@ static void imx_fim_wait_first_timestamp(void *priv)
 	struct imx_media_fim *fim = priv;
 	unsigned long ret;
 
-	ret = wait_for_completion_timeout(
-			&fim->icap_first_event,
-			msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+	ret = wait_for_completion_timeout(&fim->icap_first_event,
+					  msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
 	if (ret == 0)
 		v4l2_warn(fim->sd, "wait first icap event timeout\n");
 }
@@ -75,6 +74,8 @@ static int fim_s_ctrl(struct v4l2_ctrl *ctrl)
 		if (v4l2_fim_is_streaming(fim->core))
 			ret = -EBUSY;
 		break;
+	case V4L2_CID_IMX_FIM_ICAP_CHANNEL:
+		break;
 	default:
 		ret = -EINVAL;
 		break;
Signature · 1 line
2.53.0
#6 lore raw

Re: [PATCH 3/3] media: imx: allow setting FIM input capture channel

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Low] media: imx: unreachable ICAP_CHANNEL case in fim_s_ctrl() due to clustering --

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] media: imx: unreachable ICAP_CHANNEL case in fim_s_ctrl() due to clustering
--

commit 3f6e136c4d9e74ffb8117949ce551b405c749b3a
Author: Jiaqi Shi <[email protected]>

media: imx: allow setting FIM input capture channel

This commit adds a V4L2_CID_IMX_FIM_ICAP_CHANNEL case to fim_s_ctrl() and reformats a wait_for_completion_timeout() call.

Quoted 4 lines
drivers/staging/media/imx/imx-media-fim.c +0 -0
index b6467e29ac288..300635a3b5527 100644

[ ... ]

drivers/staging/media/imx/imx-media-fim.c static int fim_s_ctrl(struct v4l2_ctrl *ctrl) · line 81

Quoted 9 lines
drivers/staging/media/imx/imx-media-fim.c +2 -0
@@ -75,6 +74,8 @@ static int fim_s_ctrl(struct v4l2_ctrl *ctrl)
 		if (v4l2_fim_is_streaming(fim->core))
 			ret = -EBUSY;
 		break;
+	case V4L2_CID_IMX_FIM_ICAP_CHANNEL:
+		break;
 	default:
 		ret = -EINVAL;
 		break;

[Severity: Low]
Is this new V4L2_CID_IMX_FIM_ICAP_CHANNEL case reachable?

imx_media_fim_init() clusters fim->icap_ctrl[] via v4l2_ctrl_cluster(FIM_NUM_ICAP_CONTROLS, fim->icap_ctrl) with V4L2_CID_IMX_FIM_ICAP_EDGE at index 0 as the cluster master. When userspace updates any control in a cluster, try_or_set_cluster() always invokes s_ctrl on ctrl->cluster[0], so fim_s_ctrl() is always called with ctrl->id == V4L2_CID_IMX_FIM_ICAP_EDGE. Because v4l2_fim_reset() then reads both icap_edge->val and icap_chan->val in imx_fim_external_timestamp(), setting V4L2_CID_IMX_FIM_ICAP_CHANNEL was not rejected with -EINVAL, and this new case is dead code that also omits the v4l2_fim_is_streaming(fim->core) check.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3