media: imx: staging cleanup and FIM export to v4l2-core
v5 cover media: imx: staging cleanup and FIM export to v4l2-core
Changes in v5:
- Use the source-independent frame-size enumeration for legacy capture nodes as well, instead of pairing generic pixel-format enumeration with source-dependent frame-size enumeration.
- Restrict IC capture format enumeration and fallback to YUV/RGB formats; CSI capture nodes continue to support the full format set.
- Keep the remaining STD and frame-interval legacy ioctls unchanged.
- Add kernel-doc for the public FIM configuration and helper APIs, including ownership, callback context, and locking rules.
This series continues the i.MX staging media driver cleanup by removing legacy control inheritance and source-dependent format handling from capture video nodes, and exports the generic Frame Interval Monitor to v4l2-core while retaining the i.MX input capture controls locally.
The new FIM files are already covered by the existing MEDIA INPUT INFRASTRUCTURE entries for drivers/media/ and include/media/.
Validation:
- allmodconfig build and sparse
- scripts/kernel-doc -none on the FIM header and implementation
- local FIM KUnit tests, not included in this series: 13/13 passed
- local white-box i.MX capture format tests, not included: 21/21 passed
- QEMU vivid: v4l2-compliance 125/125, 30-frame streaming passed
- QEMU vim2m: v4l2-compliance 49/49, 10-frame streaming passed
- i.MX register-level and hardware DMA paths remain untested
Jiaqi Shi (2):
media: imx: remove legacy video node control and format handling
media: v4l2-core: add frame interval monitor helper
Documentation/admin-guide/media/imx.rst | 20 +- drivers/media/v4l2-core/Kconfig | 4 + drivers/media/v4l2-core/Makefile | 1 + drivers/media/v4l2-core/v4l2-fim.c | 385 +++++++++++++++++ drivers/staging/media/imx/Kconfig | 1 + drivers/staging/media/imx/TODO | 13 - drivers/staging/media/imx/imx-ic-prpencvf.c | 10 +- drivers/staging/media/imx/imx-media-capture.c | 391 +++++++----------- drivers/staging/media/imx/imx-media-csi.c | 10 +- .../staging/media/imx/imx-media-dev-common.c | 113 +---- drivers/staging/media/imx/imx-media-fim.c | 361 +++------------- drivers/staging/media/imx/imx-media-utils.c | 9 +- drivers/staging/media/imx/imx-media.h | 11 +- include/media/v4l2-fim.h | 57 +++ 14 files changed, 672 insertions(+), 714 deletions(-) create mode 100644 drivers/media/v4l2-core/v4l2-fim.c delete mode 100644 drivers/staging/media/imx/TODO create mode 100644 include/media/v4l2-fim.h
Signature · 1 line
2.53.0
PATCH v5 1/2 media: imx: remove legacy video node control and format handling
Remove control inheritance from pipeline subdevices to video nodes. Userspace should access controls through subdev nodes instead.
Use source-independent format ioctls for all capture nodes and initialize them to the existing UYVY 640x480 default instead of the source subdev active format. Keep stream-time format validation.
Update the usage documentation and remove the completed TODO entries.
--- Documentation/admin-guide/media/imx.rst | 20 +- drivers/staging/media/imx/TODO | 10 - drivers/staging/media/imx/imx-ic-prpencvf.c | 10 +- drivers/staging/media/imx/imx-media-capture.c | 391 +++++++----------- drivers/staging/media/imx/imx-media-csi.c | 10 +- .../staging/media/imx/imx-media-dev-common.c | 113 +---- drivers/staging/media/imx/imx-media-utils.c | 9 +- drivers/staging/media/imx/imx-media.h | 11 +- 8 files changed, 169 insertions(+), 405 deletions(-)
Documentation/admin-guide/media/imx.rst +8 -12
@@ -394,13 +394,9 @@ scaling up to 1024x1024, CSC, flip, and rotation: Usage Notes ----------- -To aid in configuration and for backward compatibility with V4L2 -applications that access controls only from video device nodes, the -capture device interfaces inherit controls from the active entities -in the current pipeline, so controls can be accessed either directly -from the subdev or from the active capture device interface. For -example, the FIM controls are available either from the ipuX_csiY -subdevs or from the active capture device. +Controls for pipeline entities are accessed through their subdev device +nodes. For example, the FIM controls are available from the ipuX_csiY +subdevs. The following are specific usage notes for the Sabre* reference boards:
@@ -483,7 +479,7 @@ NTSC, or 'alternate'): media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY2X8/720x480]" media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/720x480]" # Configure "ipu1_csi0 capture" interface (assumed at /dev/video4) - v4l2-ctl -d4 --set-fmt-video=field=interlaced_bt + v4l2-ctl -d4 --set-fmt-video=width=720,height=480,field=interlaced_bt Streaming can then begin on /dev/video4. The v4l2-ctl tool can also be used to select any supported YUV pixelformat on /dev/video4.
@@ -510,7 +506,7 @@ alternating fields (field type 'seq-tb' for PAL, or 'alternate'). media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x576 field:none]" media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x576 field:none]" # Configure "ipu1_ic_prpvf capture" interface (assumed at /dev/video2) - v4l2-ctl -d2 --set-fmt-video=field=none + v4l2-ctl -d2 --set-fmt-video=width=720,height=576,field=none Streaming can then begin on /dev/video2. The v4l2-ctl tool can also be used to select any supported YUV pixelformat on /dev/video2.
@@ -541,7 +537,7 @@ NTSC, or 'alternate'): media-ctl -V "'ipu1_csi0_mux':5 [fmt:UYVY2X8/720x480]" media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/720x480]" # Configure "ipu1_csi0 capture" interface (assumed at /dev/video0) - v4l2-ctl -d0 --set-fmt-video=field=interlaced_bt + v4l2-ctl -d0 --set-fmt-video=width=720,height=480,field=interlaced_bt Streaming can then begin on /dev/video0. The v4l2-ctl tool can also be used to select any supported YUV pixelformat on /dev/video0.
@@ -568,7 +564,7 @@ alternating fields (field type 'seq-tb' for PAL, or 'alternate'). media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x576 field:none]" media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x576 field:none]" # Configure "ipu1_ic_prpvf capture" interface (assumed at /dev/video2) - v4l2-ctl -d2 --set-fmt-video=field=none + v4l2-ctl -d2 --set-fmt-video=width=720,height=576,field=none Streaming can then begin on /dev/video2. The v4l2-ctl tool can also be used to select any supported YUV pixelformat on /dev/video2.
@@ -652,7 +648,7 @@ shows colorspace conversion and scaling at IC output. media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x480]" media-ctl -V "'ipu1_ic_prpenc':1 [fmt:ARGB8888_1X32/800x600]" # Set a format at the capture interface - v4l2-ctl -d /dev/video1 --set-fmt-video=pixelformat=RGB3 + v4l2-ctl -d /dev/video1 --set-fmt-video=width=800,height=600,pixelformat=RGB3 Streaming can then begin on "ipu1_ic_prpenc capture" node.
drivers/staging/media/imx/TODO +0 -10
@@ -1,13 +1,3 @@ - The Frame Interval Monitor could be exported to v4l2-core for general use. - -- This media driver supports inheriting V4L2 controls to the - video capture devices, from the subdevices in the capture device's - pipeline. The controls for each capture device are updated in the - link_notify callback when the pipeline is modified. This feature should be - removed, userspace should use the subdev-based userspace API instead. - -- Similarly to the legacy control handling, legacy format handling where - formats on the video nodes are influenced by the active format of the - connected subdev should be removed.
drivers/staging/media/imx/imx-ic-prpencvf.c +4 -6
@@ -203,7 +203,7 @@ static void prp_vb2_buf_done(struct prp_priv *priv, struct ipuv3_channel *ch) done = priv->active_vb2_buf[priv->ipu_buf_num];if (done) { - done->vbuf.field = vdev->fmt.field; + done->vbuf.field = vdev->stream_field; done->vbuf.sequence = priv->frame_sequence; vb = &done->vbuf.vb2_buf; vb->timestamp =ktime_get_ns ();
@@ -362,6 +362,7 @@ static int prp_setup_channel(struct prp_priv *priv,memset (&image,0 ,sizeof (image)); image.pix = vdev->fmt; + image.pix.field = vdev->stream_field; image.rect = vdev->compose;/*
@@ -1280,13 +1281,13 @@ static int prp_registered(struct v4l2_subdev *sd) priv->vdev =imx_media_capture_device_init (ic_priv->ipu_dev, &ic_priv->sd, -PRPENCVF_SRC_PAD , true); +PRPENCVF_SRC_PAD , true, true);if (IS_ERR (priv->vdev))return PTR_ERR (priv->vdev); ret =imx_media_capture_device_register (priv->vdev,0 );if (ret) - goto remove_vdev; +return ret; ret =prp_init_controls (priv);if (ret)
@@ -1296,8 +1297,6 @@ static int prp_registered(struct v4l2_subdev *sd) unreg_vdev:imx_media_capture_device_unregister (priv->vdev); -remove_vdev: -imx_media_capture_device_remove (priv->vdev);return ret; }
@@ -1306,7 +1305,6 @@ static void prp_unregistered(struct v4l2_subdev *sd)struct prp_priv *priv =sd_to_priv (sd);imx_media_capture_device_unregister (priv->vdev); -imx_media_capture_device_remove (priv->vdev);v4l2_ctrl_handler_free (&priv->ctrl_hdlr); }
drivers/staging/media/imx/imx-media-capture.c +140 -251
555 lines not on this page · raw message
drivers/staging/media/imx/imx-media-csi.c +4 -6
@@ -266,7 +266,7 @@ static void csi_vb2_buf_done(struct csi_priv *priv) done = priv->active_vb2_buf[priv->ipu_buf_num];if (done) { - done->vbuf.field = vdev->fmt.field; + done->vbuf.field = vdev->stream_field; done->vbuf.sequence = priv->frame_sequence; vb = &done->vbuf.vb2_buf; vb->timestamp =ktime_get_ns ();
@@ -423,6 +423,7 @@ static int csi_idmac_setup_channel(struct csi_priv *priv,memset (&image,0 ,sizeof (image)); image.pix = vdev->fmt; + image.pix.field = vdev->stream_field; image.rect = vdev->compose;csi_idmac_setup_vb2_buf (priv, phys);
@@ -1815,7 +1816,7 @@ static int csi_registered(struct v4l2_subdev *sd) } priv->vdev =imx_media_capture_device_init (priv->sd.dev, &priv->sd, -CSI_SRC_PAD_IDMAC , true); +CSI_SRC_PAD_IDMAC , true, false);if (IS_ERR (priv->vdev)) { ret =PTR_ERR (priv->vdev); goto free_fim;
@@ -1823,12 +1824,10 @@ static int csi_registered(struct v4l2_subdev *sd) ret =imx_media_capture_device_register (priv->vdev,0 );if (ret) - goto remove_vdev; + goto free_fim;return 0 ; -remove_vdev: -imx_media_capture_device_remove (priv->vdev); free_fim:if (priv->fim)imx_media_fim_free (priv->fim);
@@ -1842,7 +1841,6 @@ static void csi_unregistered(struct v4l2_subdev *sd)struct csi_priv *priv =v4l2_get_subdevdata (sd);imx_media_capture_device_unregister (priv->vdev); -imx_media_capture_device_remove (priv->vdev);if (priv->fim)imx_media_fim_free (priv->fim);
drivers/staging/media/imx/imx-media-dev-common.c +1 -112
131 lines not on this page · raw message
drivers/staging/media/imx/imx-media-utils.c +7 -2
@@ -522,6 +522,7 @@ int imx_media_mbus_fmt_to_pix_fmt(struct v4l2_pix_format *pix, {u32 width;u32 stride; +u64 sizeimage;if (!cc) { cc = imx_media_find_ipu_format(mbus->code,
@@ -562,8 +563,12 @@ int imx_media_mbus_fmt_to_pix_fmt(struct v4l2_pix_format *pix, pix->quantization = mbus->quantization; pix->field = mbus->field; pix->bytesperline = stride; - pix->sizeimage = cc->planar ? ((stride * pix->height * cc->bpp) >>3 ) : - stride * pix->height; + sizeimage = (u64 )stride * pix->height; +if (cc->planar) + sizeimage = (sizeimage * cc->bpp) >>3 ; +if (sizeimage >U32_MAX ) +return -EINVAL ; + pix->sizeimage = sizeimage;return 0 ; }
drivers/staging/media/imx/imx-media.h +5 -6
@@ -109,6 +109,8 @@ struct imx_media_video_dev {/* the user format */ struct v4l2_pix_format fmt; +/* field order resolved for the current stream */ +u32 stream_field;/* the compose rectangle */ struct v4l2_rect compose;const struct imx_media_pixfmt *cc;
@@ -125,10 +127,8 @@ static inline struct imx_media_buffer *to_imx_media_vb(struct vb2_buffer *vb) }/* -* to support control inheritance to video devices, this -* retrieves a pad's list_head of video devices that can -* be reached from the pad. Note that only the lists in -* source pads get populated, sink pads have empty lists. +* Retrieve a pad's list of video devices reachable from the pad for +* subdev event forwarding. Only source pad lists are populated. */ static inline struct list_head *to_pad_vdev_list (struct v4l2_subdev *sd,int pad_index)
@@ -270,8 +270,7 @@ int imx_media_ic_unregister(struct v4l2_subdev *sd);/* imx-media-capture.c */ struct imx_media_video_dev *imx_media_capture_device_init (struct device *dev,struct v4l2_subdev *src_sd, -int pad,bool legacy_api); -void imx_media_capture_device_remove (struct imx_media_video_dev *vdev); +int pad,bool legacy_api,bool ic_route);int imx_media_capture_device_register (struct imx_media_video_dev *vdev,u32 link_flags);void imx_media_capture_device_unregister (struct imx_media_video_dev *vdev);
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v5 1/2] media: imx: remove legacy video node control and format handling
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider: - [Medium] media: imx: capture_enum_framesizes() allows Bayer formats on IC nodes - [Medium] media: imx…
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider:
- [Medium] media: imx: capture_enum_framesizes() allows Bayer formats on IC nodes
- [Medium] media: imx: callers of imx_media_capture_device_init() still pass legacy_api
--
commit f8bbc2bd5d6e37d94f858bc1efd4d148595db2f1
Author: Jiaqi Shi <[email protected]>
media: imx: remove legacy video node control and format handling
This commit removes subdevice control inheritance on capture video nodes and switches capture format ioctls and initial format setup to be independent of the connected source subdevice's active format.
Quoted 4 lines
drivers/staging/media/imx/imx-media-capture.c +0
-0
[ ... ]
Quoted 12 lines
drivers/staging/media/imx/imx-media-capture.c +5
-1@@ -93,8 +93,12 @@ static int capture_querycap(struct file *file, void *fh,static int capture_enum_fmt_vid_cap (struct file *file,void *fh,struct v4l2_fmtdesc *f) { +struct capture_priv *priv =video_drvdata (file); +enum imx_pixfmt_sel fmt_sel = priv->ic_route ?PIXFMT_SEL_YUV_RGB : +PIXFMT_SEL_ANY ; +return imx_media_enum_pixel_formats (&f->pixelformat, f->index, -PIXFMT_SEL_ANY , f->mbus_code); + fmt_sel, f->mbus_code); }
[ ... ]
Quoted 5 lines
drivers/staging/media/imx/imx-media-capture.c +0
-1@@ -563,13 +437,13 @@ static int capture_legacy_subscribe_event(struct v4l2_fh *fh,static const struct v4l2_ioctl_ops capture_legacy_ioctl_ops = { .vidioc_querycap = capture_querycap, - .vidioc_enum_framesizes = capture_legacy_enum_framesizes,
drivers/staging/media/imx/imx-media-capture.c static int capture_legacy_subscribe_event(struct v4l2_fh *fh, · line 446
Quoted 10 lines
drivers/staging/media/imx/imx-media-capture.c +4
-3+ .vidioc_enum_framesizes = capture_enum_framesizes, .vidioc_enum_frameintervals = capture_legacy_enum_frameintervals, - .vidioc_enum_fmt_vid_cap = capture_legacy_enum_fmt_vid_cap, + .vidioc_enum_fmt_vid_cap = capture_enum_fmt_vid_cap, .vidioc_g_fmt_vid_cap = capture_g_fmt_vid_cap, - .vidioc_try_fmt_vid_cap = capture_legacy_try_fmt_vid_cap, - .vidioc_s_fmt_vid_cap = capture_legacy_s_fmt_vid_cap, + .vidioc_try_fmt_vid_cap = capture_try_fmt_vid_cap, + .vidioc_s_fmt_vid_cap = capture_s_fmt_vid_cap,
[Severity: Medium]
Should capture_enum_framesizes() and capture_legacy_enum_frameintervals() also filter pixel formats using priv->ic_route ? PIXFMT_SEL_YUV_RGB :PIXFMT_SEL_ANY?
capture_enum_fmt_vid_cap() and __capture_try_fmt() restrict IC capture nodes (ipuX_ic_prpenc and ipuX_ic_prpvf) to PIXFMT_SEL_YUV_RGB, but capture_enum_framesizes() and capture_legacy_enum_frameintervals() still look up the format with PIXFMT_SEL_ANY:
drivers/staging/media/imx/imx-media-capture.c:capture_enum_framesizes() {
if (fsize->index > 0)
return -EINVAL;
cc = imx_media_find_pixel_format(fsize->pixel_format, PIXFMT_SEL_ANY);
if (!cc)
return -EINVAL;
...
}As a result, VIDIOC_ENUM_FRAMESIZES on IC capture nodes now succeeds for Bayer and greyscale formats that VIDIOC_ENUM_FMT does not enumerate and VIDIOC_TRY_FMT rejects.
[ ... ]
Quoted 10 lines
drivers/staging/media/imx/imx-media-capture.c +1
-1@@ -960,7 +863,7 @@ EXPORT_SYMBOL_GPL(imx_media_capture_device_unregister);struct imx_media_video_dev *imx_media_capture_device_init (struct device *dev,struct v4l2_subdev *src_sd, -int pad,bool legacy_api) +int pad,bool legacy_api,bool ic_route) {struct capture_priv *priv;struct video_device *vfd;
drivers/staging/media/imx/imx-media-capture.c imx_media_capture_device_init(struct device *dev, struct v4l2_subdev *src_sd, · line 880
Quoted 5 lines
drivers/staging/media/imx/imx-media-capture.c +1
-1@@ -974,7 +877,7 @@ imx_media_capture_device_init(struct device *dev, struct v4l2_subdev *src_sd, priv->src_sd = src_sd; priv->src_sd_pad = pad; priv->dev = dev; - priv->legacy_api = legacy_api; + priv->ic_route = ic_route;
[Severity: Medium]
Should prp_registered() and csi_registered() stop passing legacy_api = true to imx_media_capture_device_init(), or should legacy_api be removed?
Both callers still pass true, so imx_media_capture_device_init() never sets V4L2_CAP_IO_MC in vfd->device_caps and always selects capture_legacy_ioctl_ops, leaving capture_ioctl_ops unreachable. Without V4L2_CAP_IO_MC, v4l_enum_fmt() in v4l2-core clears f->mbus_code before calling capture_enum_fmt_vid_cap(), which disables mbus_code format filtering, and capture_legacy_enum_frameintervals() remains coupled to the connected subdev's active format (or fails with -ENOIOCTLCMD on IC capture nodes because prp_pad_ops does not implement enum_frame_interval).
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
PATCH v5 2/2 media: v4l2-core: add frame interval monitor helper
In reply to Jiaqi ShiMove the generic frame interval monitoring implementation from i.MX to v4l2-core. The helper supports driver-selected control and event IDs and accepts EOF or external timestamps.
Keep the i.MX-specific input capture controls in the i.MX adapter, sharing the original control handler and preserving the existing UAPI and event semantics.
Select V4L2_FIM from VIDEO_IMX_MEDIA and remove the completed TODO.
--- drivers/media/v4l2-core/Kconfig | 4 + drivers/media/v4l2-core/Makefile | 1 + drivers/media/v4l2-core/v4l2-fim.c | 385 ++++++++++++++++++++++ drivers/staging/media/imx/Kconfig | 1 + drivers/staging/media/imx/TODO | 3 - drivers/staging/media/imx/imx-media-fim.c | 361 ++++---------------- include/media/v4l2-fim.h | 57 ++++ 7 files changed, 503 insertions(+), 309 deletions(-) create mode 100644 drivers/media/v4l2-core/v4l2-fim.c delete mode 100644 drivers/staging/media/imx/TODO create mode 100644 include/media/v4l2-fim.h
drivers/media/v4l2-core/Kconfig +4 -0
@@ -40,6 +40,10 @@ config VIDEO_TUNER config V4L2_JPEG_HELPER tristate +# Used by drivers that need v4l2-fim.ko +config V4L2_FIM + tristate + # Used by drivers that need v4l2-h264.ko config V4L2_H264 tristate
drivers/media/v4l2-core/Makefile +1 -0
@@ -26,6 +26,7 @@ videodev-$(CONFIG_VIDEO_V4L2_I2C) += v4l2-i2c.o obj-$(CONFIG_V4L2_ASYNC) += v4l2-async.o obj-$(CONFIG_V4L2_CCI) += v4l2-cci.o +obj-$(CONFIG_V4L2_FIM) += v4l2-fim.o obj-$(CONFIG_V4L2_FLASH_LED_CLASS) += v4l2-flash-led-class.o obj-$(CONFIG_V4L2_FWNODE) += v4l2-fwnode.o obj-$(CONFIG_V4L2_H264) += v4l2-h264.o
drivers/media/v4l2-core/v4l2-fim.c new file +385 -0
385 lines not on this page · raw message
drivers/staging/media/imx/Kconfig +1 -0
@@ -6,6 +6,7 @@ config VIDEO_IMX_MEDIA depends on VIDEO_DEV depends on IMX_IPUV3_CORE select MEDIA_CONTROLLER + select V4L2_FIM select V4L2_FWNODE select V4L2_MEM2MEM_DEV select VIDEOBUF2_DMA_CONTIG
drivers/staging/media/imx/TODO deleted +0 -3
@@ -1,3 +0,0 @@ - -- The Frame Interval Monitor could be exported to v4l2-core for - general use.
drivers/staging/media/imx/imx-media-fim.c +55 -306
468 lines not on this page · raw message
include/media/v4l2-fim.h new file +57 -0
57 lines not on this page · raw message
Signature · 1 line
2.53.0