media: imx: staging cleanup and FIM export to v4l2-core
v4 cover media: imx: staging cleanup and FIM export to v4l2-core
Changes in v4:
- Keep the user-visible format field unchanged at stream start. Resolve the actual field order into a separate stream_field so source order changes are revalidated on each stream start.
- Simplify the FIM external_timestamp callback to a single use_val flag.
- Document FIM allocation ownership, callback context, and locking rules.
The legacy_api mode remains enabled for the STD and frame-interval ioctls; removing those interfaces is outside the scope of this cleanup.
This series continues the i.MX staging media driver cleanup by removing legacy control inheritance and source-dependent format handling from capture video nodes, and exports the generic Frame Interval Monitor to v4l2-core while retaining the i.MX input capture controls locally.
Validation:
- allmodconfig build and sparse
- local FIM KUnit tests, not included in this series: 13/13 passed
- local white-box i.MX capture format tests, not included: 20/20 passed
- QEMU vivid: v4l2-compliance 125/125, 30-frame streaming passed
- QEMU vim2m: v4l2-compliance 49/49, 10-frame streaming passed
- i.MX register-level and hardware DMA paths remain untested
Jiaqi Shi (2):
media: imx: remove legacy video node control and format handling
media: v4l2-core: add frame interval monitor helper
Documentation/admin-guide/media/imx.rst | 20 +- drivers/media/v4l2-core/Kconfig | 4 + drivers/media/v4l2-core/Makefile | 1 + drivers/media/v4l2-core/v4l2-fim.c | 331 ++++++++++++++++ drivers/staging/media/imx/Kconfig | 1 + drivers/staging/media/imx/TODO | 13 - drivers/staging/media/imx/imx-ic-prpencvf.c | 10 +- drivers/staging/media/imx/imx-media-capture.c | 335 +++++++--------- drivers/staging/media/imx/imx-media-csi.c | 10 +- .../staging/media/imx/imx-media-dev-common.c | 113 +----- drivers/staging/media/imx/imx-media-fim.c | 361 +++--------------- drivers/staging/media/imx/imx-media-utils.c | 9 +- drivers/staging/media/imx/imx-media.h | 11 +- include/media/v4l2-fim.h | 49 +++ 14 files changed, 599 insertions(+), 669 deletions(-) create mode 100644 drivers/media/v4l2-core/v4l2-fim.c delete mode 100644 drivers/staging/media/imx/TODO create mode 100644 include/media/v4l2-fim.h
Signature · 1 line
2.53.0
PATCH v4 1/2 media: imx: remove legacy video node control and format handling
Remove control inheritance from pipeline subdevices to video nodes. Userspace should access controls through subdev nodes instead.
Use source-independent format ioctls for all capture nodes and initialize them to the existing UYVY 640x480 default instead of the source subdev active format. Keep stream-time format validation.
Update the usage documentation and remove the completed TODO entries.
--- Documentation/admin-guide/media/imx.rst | 20 +- drivers/staging/media/imx/TODO | 10 - drivers/staging/media/imx/imx-ic-prpencvf.c | 10 +- drivers/staging/media/imx/imx-media-capture.c | 335 +++++++----------- drivers/staging/media/imx/imx-media-csi.c | 10 +- .../staging/media/imx/imx-media-dev-common.c | 113 +----- drivers/staging/media/imx/imx-media-utils.c | 9 +- drivers/staging/media/imx/imx-media.h | 11 +- 8 files changed, 158 insertions(+), 360 deletions(-)
Documentation/admin-guide/media/imx.rst +8 -12
@@ -394,13 +394,9 @@ scaling up to 1024x1024, CSC, flip, and rotation: Usage Notes ----------- -To aid in configuration and for backward compatibility with V4L2 -applications that access controls only from video device nodes, the -capture device interfaces inherit controls from the active entities -in the current pipeline, so controls can be accessed either directly -from the subdev or from the active capture device interface. For -example, the FIM controls are available either from the ipuX_csiY -subdevs or from the active capture device. +Controls for pipeline entities are accessed through their subdev device +nodes. For example, the FIM controls are available from the ipuX_csiY +subdevs. The following are specific usage notes for the Sabre* reference boards:
@@ -483,7 +479,7 @@ NTSC, or 'alternate'): media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY2X8/720x480]" media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/720x480]" # Configure "ipu1_csi0 capture" interface (assumed at /dev/video4) - v4l2-ctl -d4 --set-fmt-video=field=interlaced_bt + v4l2-ctl -d4 --set-fmt-video=width=720,height=480,field=interlaced_bt Streaming can then begin on /dev/video4. The v4l2-ctl tool can also be used to select any supported YUV pixelformat on /dev/video4.
@@ -510,7 +506,7 @@ alternating fields (field type 'seq-tb' for PAL, or 'alternate'). media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x576 field:none]" media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x576 field:none]" # Configure "ipu1_ic_prpvf capture" interface (assumed at /dev/video2) - v4l2-ctl -d2 --set-fmt-video=field=none + v4l2-ctl -d2 --set-fmt-video=width=720,height=576,field=none Streaming can then begin on /dev/video2. The v4l2-ctl tool can also be used to select any supported YUV pixelformat on /dev/video2.
@@ -541,7 +537,7 @@ NTSC, or 'alternate'): media-ctl -V "'ipu1_csi0_mux':5 [fmt:UYVY2X8/720x480]" media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/720x480]" # Configure "ipu1_csi0 capture" interface (assumed at /dev/video0) - v4l2-ctl -d0 --set-fmt-video=field=interlaced_bt + v4l2-ctl -d0 --set-fmt-video=width=720,height=480,field=interlaced_bt Streaming can then begin on /dev/video0. The v4l2-ctl tool can also be used to select any supported YUV pixelformat on /dev/video0.
@@ -568,7 +564,7 @@ alternating fields (field type 'seq-tb' for PAL, or 'alternate'). media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x576 field:none]" media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x576 field:none]" # Configure "ipu1_ic_prpvf capture" interface (assumed at /dev/video2) - v4l2-ctl -d2 --set-fmt-video=field=none + v4l2-ctl -d2 --set-fmt-video=width=720,height=576,field=none Streaming can then begin on /dev/video2. The v4l2-ctl tool can also be used to select any supported YUV pixelformat on /dev/video2.
@@ -652,7 +648,7 @@ shows colorspace conversion and scaling at IC output. media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x480]" media-ctl -V "'ipu1_ic_prpenc':1 [fmt:ARGB8888_1X32/800x600]" # Set a format at the capture interface - v4l2-ctl -d /dev/video1 --set-fmt-video=pixelformat=RGB3 + v4l2-ctl -d /dev/video1 --set-fmt-video=width=800,height=600,pixelformat=RGB3 Streaming can then begin on "ipu1_ic_prpenc capture" node.
drivers/staging/media/imx/TODO +0 -10
@@ -1,13 +1,3 @@ - The Frame Interval Monitor could be exported to v4l2-core for general use. - -- This media driver supports inheriting V4L2 controls to the - video capture devices, from the subdevices in the capture device's - pipeline. The controls for each capture device are updated in the - link_notify callback when the pipeline is modified. This feature should be - removed, userspace should use the subdev-based userspace API instead. - -- Similarly to the legacy control handling, legacy format handling where - formats on the video nodes are influenced by the active format of the - connected subdev should be removed.
drivers/staging/media/imx/imx-ic-prpencvf.c +4 -6
@@ -203,7 +203,7 @@ static void prp_vb2_buf_done(struct prp_priv *priv, struct ipuv3_channel *ch) done = priv->active_vb2_buf[priv->ipu_buf_num];if (done) { - done->vbuf.field = vdev->fmt.field; + done->vbuf.field = vdev->stream_field; done->vbuf.sequence = priv->frame_sequence; vb = &done->vbuf.vb2_buf; vb->timestamp =ktime_get_ns ();
@@ -362,6 +362,7 @@ static int prp_setup_channel(struct prp_priv *priv,memset (&image,0 ,sizeof (image)); image.pix = vdev->fmt; + image.pix.field = vdev->stream_field; image.rect = vdev->compose;/*
@@ -1280,13 +1281,13 @@ static int prp_registered(struct v4l2_subdev *sd) priv->vdev =imx_media_capture_device_init (ic_priv->ipu_dev, &ic_priv->sd, -PRPENCVF_SRC_PAD , true); +PRPENCVF_SRC_PAD , true, true);if (IS_ERR (priv->vdev))return PTR_ERR (priv->vdev); ret =imx_media_capture_device_register (priv->vdev,0 );if (ret) - goto remove_vdev; +return ret; ret =prp_init_controls (priv);if (ret)
@@ -1296,8 +1297,6 @@ static int prp_registered(struct v4l2_subdev *sd) unreg_vdev:imx_media_capture_device_unregister (priv->vdev); -remove_vdev: -imx_media_capture_device_remove (priv->vdev);return ret; }
@@ -1306,7 +1305,6 @@ static void prp_unregistered(struct v4l2_subdev *sd)struct prp_priv *priv =sd_to_priv (sd);imx_media_capture_device_unregister (priv->vdev); -imx_media_capture_device_remove (priv->vdev);v4l2_ctrl_handler_free (&priv->ctrl_hdlr); }
drivers/staging/media/imx/imx-media-capture.c +129 -206
480 lines not on this page · raw message
drivers/staging/media/imx/imx-media-csi.c +4 -6
@@ -266,7 +266,7 @@ static void csi_vb2_buf_done(struct csi_priv *priv) done = priv->active_vb2_buf[priv->ipu_buf_num];if (done) { - done->vbuf.field = vdev->fmt.field; + done->vbuf.field = vdev->stream_field; done->vbuf.sequence = priv->frame_sequence; vb = &done->vbuf.vb2_buf; vb->timestamp =ktime_get_ns ();
@@ -423,6 +423,7 @@ static int csi_idmac_setup_channel(struct csi_priv *priv,memset (&image,0 ,sizeof (image)); image.pix = vdev->fmt; + image.pix.field = vdev->stream_field; image.rect = vdev->compose;csi_idmac_setup_vb2_buf (priv, phys);
@@ -1815,7 +1816,7 @@ static int csi_registered(struct v4l2_subdev *sd) } priv->vdev =imx_media_capture_device_init (priv->sd.dev, &priv->sd, -CSI_SRC_PAD_IDMAC , true); +CSI_SRC_PAD_IDMAC , true, false);if (IS_ERR (priv->vdev)) { ret =PTR_ERR (priv->vdev); goto free_fim;
@@ -1823,12 +1824,10 @@ static int csi_registered(struct v4l2_subdev *sd) ret =imx_media_capture_device_register (priv->vdev,0 );if (ret) - goto remove_vdev; + goto free_fim;return 0 ; -remove_vdev: -imx_media_capture_device_remove (priv->vdev); free_fim:if (priv->fim)imx_media_fim_free (priv->fim);
@@ -1842,7 +1841,6 @@ static void csi_unregistered(struct v4l2_subdev *sd)struct csi_priv *priv =v4l2_get_subdevdata (sd);imx_media_capture_device_unregister (priv->vdev); -imx_media_capture_device_remove (priv->vdev);if (priv->fim)imx_media_fim_free (priv->fim);
drivers/staging/media/imx/imx-media-dev-common.c +1 -112
131 lines not on this page · raw message
drivers/staging/media/imx/imx-media-utils.c +7 -2
@@ -522,6 +522,7 @@ int imx_media_mbus_fmt_to_pix_fmt(struct v4l2_pix_format *pix, {u32 width;u32 stride; +u64 sizeimage;if (!cc) { cc = imx_media_find_ipu_format(mbus->code,
@@ -562,8 +563,12 @@ int imx_media_mbus_fmt_to_pix_fmt(struct v4l2_pix_format *pix, pix->quantization = mbus->quantization; pix->field = mbus->field; pix->bytesperline = stride; - pix->sizeimage = cc->planar ? ((stride * pix->height * cc->bpp) >>3 ) : - stride * pix->height; + sizeimage = (u64 )stride * pix->height; +if (cc->planar) + sizeimage = (sizeimage * cc->bpp) >>3 ; +if (sizeimage >U32_MAX ) +return -EINVAL ; + pix->sizeimage = sizeimage;return 0 ; }
drivers/staging/media/imx/imx-media.h +5 -6
@@ -109,6 +109,8 @@ struct imx_media_video_dev {/* the user format */ struct v4l2_pix_format fmt; +/* field order resolved for the current stream */ +u32 stream_field;/* the compose rectangle */ struct v4l2_rect compose;const struct imx_media_pixfmt *cc;
@@ -125,10 +127,8 @@ static inline struct imx_media_buffer *to_imx_media_vb(struct vb2_buffer *vb) }/* -* to support control inheritance to video devices, this -* retrieves a pad's list_head of video devices that can -* be reached from the pad. Note that only the lists in -* source pads get populated, sink pads have empty lists. +* Retrieve a pad's list of video devices reachable from the pad for +* subdev event forwarding. Only source pad lists are populated. */ static inline struct list_head *to_pad_vdev_list (struct v4l2_subdev *sd,int pad_index)
@@ -270,8 +270,7 @@ int imx_media_ic_unregister(struct v4l2_subdev *sd);/* imx-media-capture.c */ struct imx_media_video_dev *imx_media_capture_device_init (struct device *dev,struct v4l2_subdev *src_sd, -int pad,bool legacy_api); -void imx_media_capture_device_remove (struct imx_media_video_dev *vdev); +int pad,bool legacy_api,bool ic_route);int imx_media_capture_device_register (struct imx_media_video_dev *vdev,u32 link_flags);void imx_media_capture_device_unregister (struct imx_media_video_dev *vdev);
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v4 1/2] media: imx: remove legacy video node control and format handling
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Medium] media: imx: callers of imx_media_capture_device_init() still pass legacy_api --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] media: imx: callers of imx_media_capture_device_init() still pass legacy_api
--
commit 38a236574cfd2ca3669e9b8ee53cd0ec8b2e47d4
Author: Jiaqi Shi <[email protected]>
media: imx: remove legacy video node control and format handling
This commit removes V4L2 control inheritance from pipeline subdevices to capture video nodes and switches capture video nodes to source-independent format ioctls and default formats.
Quoted 4 lines
drivers/staging/media/imx/imx-ic-prpencvf.c +0
-0
[ ... ]
Quoted 6 lines
drivers/staging/media/imx/imx-ic-prpencvf.c +1
-1@@ -1280,13 +1281,13 @@ static int prp_registered(struct v4l2_subdev *sd) priv->vdev =imx_media_capture_device_init (ic_priv->ipu_dev, &ic_priv->sd, -PRPENCVF_SRC_PAD , true); +PRPENCVF_SRC_PAD , true, true);
[ ... ]
Quoted 4 lines
drivers/staging/media/imx/imx-media-capture.c +0
-0
[ ... ]
drivers/staging/media/imx/imx-media-capture.c static const struct v4l2_ioctl_ops capture_legacy_ioctl_ops = { · line 480
Quoted 11 lines
drivers/staging/media/imx/imx-media-capture.c +3
-3@@ -566,10 +474,10 @@ static const struct v4l2_ioctl_ops capture_legacy_ioctl_ops = { .vidioc_enum_framesizes = capture_legacy_enum_framesizes, .vidioc_enum_frameintervals = capture_legacy_enum_frameintervals, - .vidioc_enum_fmt_vid_cap = capture_legacy_enum_fmt_vid_cap, + .vidioc_enum_fmt_vid_cap = capture_enum_fmt_vid_cap, .vidioc_g_fmt_vid_cap = capture_g_fmt_vid_cap, - .vidioc_try_fmt_vid_cap = capture_legacy_try_fmt_vid_cap, - .vidioc_s_fmt_vid_cap = capture_legacy_s_fmt_vid_cap, + .vidioc_try_fmt_vid_cap = capture_try_fmt_vid_cap, + .vidioc_s_fmt_vid_cap = capture_s_fmt_vid_cap,
[Severity: Medium]
Should prp_registered() and csi_registered() pass legacy_api = false to imx_media_capture_device_init() (or should the legacy_api path be removed)?
Because both callers still pass legacy_api = true, capture_legacy_ioctl_ops is always used and V4L2_CAP_IO_MC is never set, leaving capture_ioctl_ops, capture_enum_framesizes() and capture_subscribe_event() unreachable and causing v4l_enum_fmt() in v4l2-core to clear f->mbus_code before capture_enum_fmt_vid_cap() runs.
In addition, capture_legacy_ioctl_ops pairs capture_enum_fmt_vid_cap() with capture_legacy_enum_framesizes(), which queries the connected subdev's active enum_frame_size op using cc->codes[0] (or 0 for in-memory-only planar formats). On ipuX_ic_prpenc and ipuX_ic_prpvf capture nodes, prp_enum_frame_size() only accepts IPU-internal media bus codes, so VIDIOC_ENUM_FRAMESIZES now fails with -EINVAL for 22 of the 23 formats enumerated by VIDIOC_ENUM_FMT (including the default UYVY), and on ipuX_csiY capture nodes frame size enumeration remains coupled to the CSI subdev's active format.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
PATCH v4 2/2 media: v4l2-core: add frame interval monitor helper
In reply to Jiaqi ShiMove the generic frame interval monitoring implementation from i.MX to v4l2-core. The helper supports driver-selected control and event IDs and accepts EOF or external timestamps.
Keep the i.MX-specific input capture controls in the i.MX adapter, sharing the original control handler and preserving the existing UAPI and event semantics.
Select V4L2_FIM from VIDEO_IMX_MEDIA and remove the completed TODO.
--- drivers/media/v4l2-core/Kconfig | 4 + drivers/media/v4l2-core/Makefile | 1 + drivers/media/v4l2-core/v4l2-fim.c | 331 ++++++++++++++++++++ drivers/staging/media/imx/Kconfig | 1 + drivers/staging/media/imx/TODO | 3 - drivers/staging/media/imx/imx-media-fim.c | 361 ++++------------------ include/media/v4l2-fim.h | 49 +++ 7 files changed, 441 insertions(+), 309 deletions(-) create mode 100644 drivers/media/v4l2-core/v4l2-fim.c delete mode 100644 drivers/staging/media/imx/TODO create mode 100644 include/media/v4l2-fim.h
drivers/media/v4l2-core/Kconfig +4 -0
@@ -40,6 +40,10 @@ config VIDEO_TUNER config V4L2_JPEG_HELPER tristate +# Used by drivers that need v4l2-fim.ko +config V4L2_FIM + tristate + # Used by drivers that need v4l2-h264.ko config V4L2_H264 tristate
drivers/media/v4l2-core/Makefile +1 -0
@@ -26,6 +26,7 @@ videodev-$(CONFIG_VIDEO_V4L2_I2C) += v4l2-i2c.o obj-$(CONFIG_V4L2_ASYNC) += v4l2-async.o obj-$(CONFIG_V4L2_CCI) += v4l2-cci.o +obj-$(CONFIG_V4L2_FIM) += v4l2-fim.o obj-$(CONFIG_V4L2_FLASH_LED_CLASS) += v4l2-flash-led-class.o obj-$(CONFIG_V4L2_FWNODE) += v4l2-fwnode.o obj-$(CONFIG_V4L2_H264) += v4l2-h264.o
drivers/media/v4l2-core/v4l2-fim.c new file +331 -0
331 lines not on this page · raw message
drivers/staging/media/imx/Kconfig +1 -0
@@ -6,6 +6,7 @@ config VIDEO_IMX_MEDIA depends on VIDEO_DEV depends on IMX_IPUV3_CORE select MEDIA_CONTROLLER + select V4L2_FIM select V4L2_FWNODE select V4L2_MEM2MEM_DEV select VIDEOBUF2_DMA_CONTIG
drivers/staging/media/imx/TODO deleted +0 -3
@@ -1,3 +0,0 @@ - -- The Frame Interval Monitor could be exported to v4l2-core for - general use.
drivers/staging/media/imx/imx-media-fim.c +55 -306
468 lines not on this page · raw message
include/media/v4l2-fim.h new file +49 -0
49 lines not on this page · raw message
Signature · 1 line
2.53.0