media: imx: staging cleanup and FIM export to v4l2-core
v3 cover media: imx: staging cleanup and FIM export to v4l2-core
Changes in v3:
- Add the missing struct v4l2_ctrl forward declaration to v4l2-fim.h.
- Validate exact media-bus-code/pixel-format pairs without falling back to another format, while preserving the intentional Y10/Y12 to GREY clamping supported by the CSI driver.
- Use 64-bit arithmetic for image size calculation and reduce the height when necessary to fit the 32-bit sizeimage field instead of failing.
- Sanitize ANY/invalid field values and validate field compatibility with the source at stream start, including IDMAC interweave field ordering.
- Default RGB formats without a media bus code to full-range quantization.
- Keep legacy_api enabled for the remaining STD and frame-interval ioctls; removing those interfaces is outside the scope of this cleanup.
This series continues the i.MX staging media driver cleanup by removing legacy control inheritance and source-dependent format handling from capture video nodes, and exports the generic Frame Interval Monitor to v4l2-core while retaining the i.MX input capture controls locally.
Validation:
- allmodconfig build and sparse
- local FIM KUnit tests, not included in this series: 13/13 passed
- local white-box i.MX capture format tests, not included: 20/20 passed
- QEMU vivid: v4l2-compliance 125/125, 30-frame streaming passed
- QEMU vim2m: v4l2-compliance 49/49, 10-frame streaming passed
- i.MX register-level and hardware DMA paths remain untested
Jiaqi Shi (2):
media: imx: remove legacy video node control and format handling
media: v4l2-core: add frame interval monitor helper
Documentation/admin-guide/media/imx.rst | 20 +- drivers/media/v4l2-core/Kconfig | 4 + drivers/media/v4l2-core/Makefile | 1 + drivers/media/v4l2-core/v4l2-fim.c | 331 ++++++++++++++++ drivers/staging/media/imx/Kconfig | 1 + drivers/staging/media/imx/TODO | 13 - drivers/staging/media/imx/imx-ic-prpencvf.c | 7 +- drivers/staging/media/imx/imx-media-capture.c | 326 ++++++---------- drivers/staging/media/imx/imx-media-csi.c | 7 +- .../staging/media/imx/imx-media-dev-common.c | 113 +----- drivers/staging/media/imx/imx-media-fim.c | 355 +++--------------- drivers/staging/media/imx/imx-media-utils.c | 9 +- drivers/staging/media/imx/imx-media.h | 9 +- include/media/v4l2-fim.h | 41 ++ 14 files changed, 573 insertions(+), 664 deletions(-) create mode 100644 drivers/media/v4l2-core/v4l2-fim.c delete mode 100644 drivers/staging/media/imx/TODO create mode 100644 include/media/v4l2-fim.h
Signature · 1 line
2.53.0
PATCH v3 1/2 media: imx: remove legacy video node control and format handling
Remove control inheritance from pipeline subdevices to video nodes. Userspace should access controls through subdev nodes instead.
Use source-independent format ioctls for all capture nodes and initialize them to the existing UYVY 640x480 default instead of the source subdev active format. Keep stream-time format validation.
Update the usage documentation and remove the completed TODO entries.
--- Documentation/admin-guide/media/imx.rst | 20 +- drivers/staging/media/imx/TODO | 10 - drivers/staging/media/imx/imx-ic-prpencvf.c | 7 +- drivers/staging/media/imx/imx-media-capture.c | 326 +++++++----------- drivers/staging/media/imx/imx-media-csi.c | 7 +- .../staging/media/imx/imx-media-dev-common.c | 113 +----- drivers/staging/media/imx/imx-media-utils.c | 9 +- drivers/staging/media/imx/imx-media.h | 9 +- 8 files changed, 143 insertions(+), 358 deletions(-)
Documentation/admin-guide/media/imx.rst +8 -12
@@ -394,13 +394,9 @@ scaling up to 1024x1024, CSC, flip, and rotation: Usage Notes ----------- -To aid in configuration and for backward compatibility with V4L2 -applications that access controls only from video device nodes, the -capture device interfaces inherit controls from the active entities -in the current pipeline, so controls can be accessed either directly -from the subdev or from the active capture device interface. For -example, the FIM controls are available either from the ipuX_csiY -subdevs or from the active capture device. +Controls for pipeline entities are accessed through their subdev device +nodes. For example, the FIM controls are available from the ipuX_csiY +subdevs. The following are specific usage notes for the Sabre* reference boards:
@@ -483,7 +479,7 @@ NTSC, or 'alternate'): media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY2X8/720x480]" media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/720x480]" # Configure "ipu1_csi0 capture" interface (assumed at /dev/video4) - v4l2-ctl -d4 --set-fmt-video=field=interlaced_bt + v4l2-ctl -d4 --set-fmt-video=width=720,height=480,field=interlaced_bt Streaming can then begin on /dev/video4. The v4l2-ctl tool can also be used to select any supported YUV pixelformat on /dev/video4.
@@ -510,7 +506,7 @@ alternating fields (field type 'seq-tb' for PAL, or 'alternate'). media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x576 field:none]" media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x576 field:none]" # Configure "ipu1_ic_prpvf capture" interface (assumed at /dev/video2) - v4l2-ctl -d2 --set-fmt-video=field=none + v4l2-ctl -d2 --set-fmt-video=width=720,height=576,field=none Streaming can then begin on /dev/video2. The v4l2-ctl tool can also be used to select any supported YUV pixelformat on /dev/video2.
@@ -541,7 +537,7 @@ NTSC, or 'alternate'): media-ctl -V "'ipu1_csi0_mux':5 [fmt:UYVY2X8/720x480]" media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/720x480]" # Configure "ipu1_csi0 capture" interface (assumed at /dev/video0) - v4l2-ctl -d0 --set-fmt-video=field=interlaced_bt + v4l2-ctl -d0 --set-fmt-video=width=720,height=480,field=interlaced_bt Streaming can then begin on /dev/video0. The v4l2-ctl tool can also be used to select any supported YUV pixelformat on /dev/video0.
@@ -568,7 +564,7 @@ alternating fields (field type 'seq-tb' for PAL, or 'alternate'). media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x576 field:none]" media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x576 field:none]" # Configure "ipu1_ic_prpvf capture" interface (assumed at /dev/video2) - v4l2-ctl -d2 --set-fmt-video=field=none + v4l2-ctl -d2 --set-fmt-video=width=720,height=576,field=none Streaming can then begin on /dev/video2. The v4l2-ctl tool can also be used to select any supported YUV pixelformat on /dev/video2.
@@ -652,7 +648,7 @@ shows colorspace conversion and scaling at IC output. media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x480]" media-ctl -V "'ipu1_ic_prpenc':1 [fmt:ARGB8888_1X32/800x600]" # Set a format at the capture interface - v4l2-ctl -d /dev/video1 --set-fmt-video=pixelformat=RGB3 + v4l2-ctl -d /dev/video1 --set-fmt-video=width=800,height=600,pixelformat=RGB3 Streaming can then begin on "ipu1_ic_prpenc capture" node.
drivers/staging/media/imx/TODO +0 -10
@@ -1,13 +1,3 @@ - The Frame Interval Monitor could be exported to v4l2-core for general use. - -- This media driver supports inheriting V4L2 controls to the - video capture devices, from the subdevices in the capture device's - pipeline. The controls for each capture device are updated in the - link_notify callback when the pipeline is modified. This feature should be - removed, userspace should use the subdev-based userspace API instead. - -- Similarly to the legacy control handling, legacy format handling where - formats on the video nodes are influenced by the active format of the - connected subdev should be removed.
drivers/staging/media/imx/imx-ic-prpencvf.c +2 -5
@@ -1280,13 +1280,13 @@ static int prp_registered(struct v4l2_subdev *sd) priv->vdev =imx_media_capture_device_init (ic_priv->ipu_dev, &ic_priv->sd, -PRPENCVF_SRC_PAD , true); +PRPENCVF_SRC_PAD , true, true);if (IS_ERR (priv->vdev))return PTR_ERR (priv->vdev); ret =imx_media_capture_device_register (priv->vdev,0 );if (ret) - goto remove_vdev; +return ret; ret =prp_init_controls (priv);if (ret)
@@ -1296,8 +1296,6 @@ static int prp_registered(struct v4l2_subdev *sd) unreg_vdev:imx_media_capture_device_unregister (priv->vdev); -remove_vdev: -imx_media_capture_device_remove (priv->vdev);return ret; }
@@ -1306,7 +1304,6 @@ static void prp_unregistered(struct v4l2_subdev *sd)struct prp_priv *priv =sd_to_priv (sd);imx_media_capture_device_unregister (priv->vdev); -imx_media_capture_device_remove (priv->vdev);v4l2_ctrl_handler_free (&priv->ctrl_hdlr); }
drivers/staging/media/imx/imx-media-capture.c +120 -206
468 lines not on this page · raw message
drivers/staging/media/imx/imx-media-csi.c +2 -5
@@ -1815,7 +1815,7 @@ static int csi_registered(struct v4l2_subdev *sd) } priv->vdev =imx_media_capture_device_init (priv->sd.dev, &priv->sd, -CSI_SRC_PAD_IDMAC , true); +CSI_SRC_PAD_IDMAC , true, false);if (IS_ERR (priv->vdev)) { ret =PTR_ERR (priv->vdev); goto free_fim;
@@ -1823,12 +1823,10 @@ static int csi_registered(struct v4l2_subdev *sd) ret =imx_media_capture_device_register (priv->vdev,0 );if (ret) - goto remove_vdev; + goto free_fim;return 0 ; -remove_vdev: -imx_media_capture_device_remove (priv->vdev); free_fim:if (priv->fim)imx_media_fim_free (priv->fim);
@@ -1842,7 +1840,6 @@ static void csi_unregistered(struct v4l2_subdev *sd)struct csi_priv *priv =v4l2_get_subdevdata (sd);imx_media_capture_device_unregister (priv->vdev); -imx_media_capture_device_remove (priv->vdev);if (priv->fim)imx_media_fim_free (priv->fim);
drivers/staging/media/imx/imx-media-dev-common.c +1 -112
131 lines not on this page · raw message
drivers/staging/media/imx/imx-media-utils.c +7 -2
@@ -522,6 +522,7 @@ int imx_media_mbus_fmt_to_pix_fmt(struct v4l2_pix_format *pix, {u32 width;u32 stride; +u64 sizeimage;if (!cc) { cc = imx_media_find_ipu_format(mbus->code,
@@ -562,8 +563,12 @@ int imx_media_mbus_fmt_to_pix_fmt(struct v4l2_pix_format *pix, pix->quantization = mbus->quantization; pix->field = mbus->field; pix->bytesperline = stride; - pix->sizeimage = cc->planar ? ((stride * pix->height * cc->bpp) >>3 ) : - stride * pix->height; + sizeimage = (u64 )stride * pix->height; +if (cc->planar) + sizeimage = (sizeimage * cc->bpp) >>3 ; +if (sizeimage >U32_MAX ) +return -EINVAL ; + pix->sizeimage = sizeimage;return 0 ; }
drivers/staging/media/imx/imx-media.h +3 -6
@@ -125,10 +125,8 @@ static inline struct imx_media_buffer *to_imx_media_vb(struct vb2_buffer *vb) }/* -* to support control inheritance to video devices, this -* retrieves a pad's list_head of video devices that can -* be reached from the pad. Note that only the lists in -* source pads get populated, sink pads have empty lists. +* Retrieve a pad's list of video devices reachable from the pad for +* subdev event forwarding. Only source pad lists are populated. */ static inline struct list_head *to_pad_vdev_list (struct v4l2_subdev *sd,int pad_index)
@@ -270,8 +268,7 @@ int imx_media_ic_unregister(struct v4l2_subdev *sd);/* imx-media-capture.c */ struct imx_media_video_dev *imx_media_capture_device_init (struct device *dev,struct v4l2_subdev *src_sd, -int pad,bool legacy_api); -void imx_media_capture_device_remove (struct imx_media_video_dev *vdev); +int pad,bool legacy_api,bool ic_route);int imx_media_capture_device_register (struct imx_media_video_dev *vdev,u32 link_flags);void imx_media_capture_device_unregister (struct imx_media_video_dev *vdev);
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v3 1/2] media: imx: remove legacy video node control and format handling
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider: - [Medium] media: imx: capture_validate_field() mutates vdev.fmt.field on stream start - [Medium] media: imx…
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider:
- [Medium] media: imx: capture_validate_field() mutates vdev.fmt.field on stream start
- [Medium] media: imx: callers of imx_media_capture_device_init() still pass legacy_api
--
commit f4224bc45a1438de57dbc43b915f511c93024a3e
Author: Jiaqi Shi <[email protected]>
media: imx: remove legacy video node control and format handling
This commit removes subdev control inheritance from capture video nodes and switches capture format handling and initialization to source-independent format ioctls while keeping stream-time format validation.
Quoted 4 lines
drivers/staging/media/imx/imx-media-capture.c +0
-0
[ ... ]
drivers/staging/media/imx/imx-media-capture.c static const struct v4l2_ioctl_ops capture_legacy_ioctl_ops = { · line 479
Quoted 11 lines
drivers/staging/media/imx/imx-media-capture.c +3
-3@@ -566,10 +473,10 @@ static const struct v4l2_ioctl_ops capture_legacy_ioctl_ops = { .vidioc_enum_framesizes = capture_legacy_enum_framesizes, .vidioc_enum_frameintervals = capture_legacy_enum_frameintervals, - .vidioc_enum_fmt_vid_cap = capture_legacy_enum_fmt_vid_cap, + .vidioc_enum_fmt_vid_cap = capture_enum_fmt_vid_cap, .vidioc_g_fmt_vid_cap = capture_g_fmt_vid_cap, - .vidioc_try_fmt_vid_cap = capture_legacy_try_fmt_vid_cap, - .vidioc_s_fmt_vid_cap = capture_legacy_s_fmt_vid_cap, + .vidioc_try_fmt_vid_cap = capture_try_fmt_vid_cap, + .vidioc_s_fmt_vid_cap = capture_s_fmt_vid_cap,
[Severity: Medium]
Should the callers of imx_media_capture_device_init() stop setting legacy_api to true now that video node format handling is decoupled from the source subdev?
Both in-tree callers (prp_registered() and csi_registered()) still pass legacy_api = true, which leaves V4L2_CAP_IO_MC unset on every capture device (so v4l_enum_fmt() zeroes f->mbus_code before calling capture_enum_fmt_vid_cap()) and leaves capture_ioctl_ops, capture_enum_framesizes(), and capture_subscribe_event() unused.
In addition, pairing capture_enum_fmt_vid_cap() with capture_legacy_enum_framesizes() breaks VIDIOC_ENUM_FRAMESIZES on the IC capture nodes: capture_enum_fmt_vid_cap() enumerates all non-IPU formats in pixel_formats[], and capture_legacy_enum_framesizes() forwards their media bus code (or 0 for in-memory-only formats) to prp_enum_frame_size(), which rejects any non-IPU media bus code with -EINVAL.
Quoted 17 lines
drivers/staging/media/imx/imx-media-capture.c +13
-0@@ -672,6 +579,44 @@ static void capture_buf_queue(struct vb2_buffer *vb)spin_unlock_irqrestore (&priv->q_lock, flags); } +static int capture_validate_field (struct capture_priv *priv, +const struct v4l2_mbus_framefmt *src) +{ +struct v4l2_pix_format *pix = &priv->vdev.fmt; + +switch (pix->field) { +case V4L2_FIELD_INTERLACED : +switch (src->field) { +case V4L2_FIELD_SEQ_TB : + pix->field =V4L2_FIELD_INTERLACED_TB ; +return 0 ; +case V4L2_FIELD_SEQ_BT : + pix->field =V4L2_FIELD_INTERLACED_BT ;
drivers/staging/media/imx/imx-media-capture.c static void capture_buf_queue(struct vb2_buffer *vb) · line 604
Quoted 10 lines
drivers/staging/media/imx/imx-media-capture.c +10
-0+return 0 ; +case V4L2_FIELD_INTERLACED : +return 0 ; +case V4L2_FIELD_INTERLACED_TB : +case V4L2_FIELD_INTERLACED_BT : + pix->field = src->field; +return 0 ; +default : +return -EPIPE ; + }
[Severity: Medium]
Should capture_validate_field() avoid overwriting priv->vdev.fmt.field during VIDIOC_STREAMON?
When userspace configures V4L2_FIELD_INTERLACED via VIDIOC_S_FMT, capture_start_streaming() -> capture_validate_fmt() -> capture_validate_field() mutates priv->vdev.fmt.field in place to V4L2_FIELD_INTERLACED_TB or V4L2_FIELD_INTERLACED_BT. This changes the format reported by VIDIOC_G_FMT (even if imx_media_pipeline_set_stream()
subsequently fails), and if the upstream subdev field order is later
switched between top-bottom and bottom-top without another VIDIOC_S_FMT, a subsequent VIDIOC_STREAMON will validate against the overwritten field order and fail with -EPIPE.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
PATCH v3 2/2 media: v4l2-core: add frame interval monitor helper
In reply to Jiaqi ShiMove the generic frame interval monitoring implementation from i.MX to v4l2-core. The helper supports driver-selected control and event IDs and accepts EOF or external timestamps.
Keep the i.MX-specific input capture controls in the i.MX adapter, sharing the original control handler and preserving the existing UAPI and event semantics.
Select V4L2_FIM from VIDEO_IMX_MEDIA and remove the completed TODO.
--- drivers/media/v4l2-core/Kconfig | 4 + drivers/media/v4l2-core/Makefile | 1 + drivers/media/v4l2-core/v4l2-fim.c | 331 ++++++++++++++++++++ drivers/staging/media/imx/Kconfig | 1 + drivers/staging/media/imx/TODO | 3 - drivers/staging/media/imx/imx-media-fim.c | 355 ++++------------------ include/media/v4l2-fim.h | 41 +++ 7 files changed, 430 insertions(+), 306 deletions(-) create mode 100644 drivers/media/v4l2-core/v4l2-fim.c delete mode 100644 drivers/staging/media/imx/TODO create mode 100644 include/media/v4l2-fim.h
drivers/media/v4l2-core/Kconfig +4 -0
@@ -40,6 +40,10 @@ config VIDEO_TUNER config V4L2_JPEG_HELPER tristate +# Used by drivers that need v4l2-fim.ko +config V4L2_FIM + tristate + # Used by drivers that need v4l2-h264.ko config V4L2_H264 tristate
drivers/media/v4l2-core/Makefile +1 -0
@@ -26,6 +26,7 @@ videodev-$(CONFIG_VIDEO_V4L2_I2C) += v4l2-i2c.o obj-$(CONFIG_V4L2_ASYNC) += v4l2-async.o obj-$(CONFIG_V4L2_CCI) += v4l2-cci.o +obj-$(CONFIG_V4L2_FIM) += v4l2-fim.o obj-$(CONFIG_V4L2_FLASH_LED_CLASS) += v4l2-flash-led-class.o obj-$(CONFIG_V4L2_FWNODE) += v4l2-fwnode.o obj-$(CONFIG_V4L2_H264) += v4l2-h264.o
drivers/media/v4l2-core/v4l2-fim.c new file +331 -0
331 lines not on this page · raw message
drivers/staging/media/imx/Kconfig +1 -0
@@ -6,6 +6,7 @@ config VIDEO_IMX_MEDIA depends on VIDEO_DEV depends on IMX_IPUV3_CORE select MEDIA_CONTROLLER + select V4L2_FIM select V4L2_FWNODE select V4L2_MEM2MEM_DEV select VIDEOBUF2_DMA_CONTIG
drivers/staging/media/imx/TODO deleted +0 -3
@@ -1,3 +0,0 @@ - -- The Frame Interval Monitor could be exported to v4l2-core for - general use.
drivers/staging/media/imx/imx-media-fim.c +52 -303
465 lines not on this page · raw message
include/media/v4l2-fim.h new file +41 -0
41 lines not on this page · raw message
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v3 2/2] media: v4l2-core: add frame interval monitor helper
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Low] media: v4l2-core: missing kernel-doc for new v4l2-fim public API --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] media: v4l2-core: missing kernel-doc for new v4l2-fim public API --
commit 70fa26e77e3e68f76d9d83df7734532469e6749d
Author: Jiaqi Shi <[email protected]>
media: v4l2-core: add frame interval monitor helper
This commit moves the generic frame interval monitoring implementation from the i.MX staging media driver to v4l2-core as a reusable helper module.
Quoted 6 lines
include/media/v4l2-fim.h new file +0
-0@@ -0,0 +1,41 @@
[ ... ]
Quoted 11 lines
include/media/v4l2-fim.h new file +11
-0+#define V4L2_FIM_NUM_CONTROLS 5 + +struct v4l2_fim_config { +u32 ctrl_base; +u32 event_type; +void *priv; +bool (*external_timestamp)(void *priv,bool curval,bool use_val); +void (*wait_first_timestamp)(void *priv); +}; + +struct v4l2_fim *v4l2_fim_init(struct v4l2_subdev *sd,
include/media/v4l2-fim.h line 41
Quoted 10 lines
include/media/v4l2-fim.h new file +10
-0+struct v4l2_ctrl_handler *hdlr, + const struct v4l2_fim_config *config); +void v4l2_fim_reset (struct v4l2_fim *fim,bool curval,struct v4l2_ctrl *ctrl); +void v4l2_fim_set_stream (struct v4l2_fim *fim, +const struct v4l2_fract *frame_interval,bool on); +void v4l2_fim_eof_monitor (struct v4l2_fim *fim,ktime_t timestamp); +void v4l2_fim_monitor_timestamp (struct v4l2_fim *fim,ktime_t timestamp); +bool v4l2_fim_is_streaming (struct v4l2_fim *fim); + +#endif
[Severity: Low]
This isn't a bug, but the new public v4l2-core helper API has no kernel-doc comments describing allocation, cleanup, or locking expectations:
- v4l2_fim_init() allocates struct v4l2_fim with devm_kzalloc(sd->dev, ...) and adds V4L2_FIM_NUM_CONTROLS controls starting at config->ctrl_base to hdlr, which the caller remains responsible for freeing with v4l2_ctrl_handler_free().
- config->external_timestamp() is called under spin_lock_irqsave(&fim->lock) with the control handler lock held, while config->wait_first_timestamp() is called from process context in v4l2_fim_set_stream() with v4l2_ctrl_lock() held.
- v4l2_fim_set_stream() acquires v4l2_ctrl_lock(), v4l2_fim_reset() and v4l2_fim_is_streaming() expect the caller to hold the control handler lock, and v4l2_fim_eof_monitor() / v4l2_fim_monitor_timestamp() acquire spin_lock_irqsave(&fim->lock) and may be called from interrupt context.
Also, reset_fim() invokes config->external_timestamp(fim->config.priv, curval, !curval && !core_active); passing both curval and use_val is redundant since the callback only needs a single boolean indicating whether to read uncommitted control->val or committed control->cur.val.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2