Skip to contentSkip to conversation

media: v4l2: Add extended fmt and buffer ioctls

RFC PATCH In review Boris Brezillon 6 participants 19 messages Oct 8, 2019 – Mar 10, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v2 cover media: v4l2: Add extended fmt and buffer ioctls

Hello,

This RFC follows the discussion started by Hans [1] a few months back. It does not try to address all the problem reported in this thread but instead focuses on the FMT and BUF(S) ioctls.

Note that my primary goal is to unify handling for multiplanar and singleplanar formats and extend things to support the "single dmabuf storing all pixel planes" issue.

This version received a bit more testing than the previous one (added new tests to v4l2-compliance [2] to make sure EXT ioctls work as expected and also checked that !ext -> ext wrappers work correctly by running the old tests). Note that I'm not planning to post those v4l-utils patches on the ML until we've settled down on the userspace API, unless I'm explicitly asked to do so.

Right now I'm focusing on the case I was primarily interested in:
single dmabuf storing all pixel planes (each being at a different offset), and it seems that patching the VB2 core to support that is not a trivial task.

So here are a few questions for V4L/DMABUF experts:
- Can the same dmabuf be mapped several times. I think it's okay apart from the extra/needless time spent doing cache maintenance operations, but there might be issues if an IOMMU is involved (duplicate mappings?). If it's not okay, then we need to find a solution to only attach/map the DMABUF once when it's used for several planes (this is what I tried to do here [3], but I'm not entirely happy with the implementation and started to investigate another approach here [4]).
- How should we pass the offset to drivers that were previously using the get_cookie() (or the dma_sg wrapper) to retrieve an sg table. Adding the offset to the dma_addr or vaddr for vmalloc or dma-contig case can be done in the core, but for an sg-table it's a bit more complicated. Should drivers access this piece of information directly from vb2_plane->dbuf_offset? And in that case, how do we make sure drivers don't simply ignore the offset and assume it's always zero?

Few words about the feedback I got from Brian and Nicolas on my v1:

- modifier field has been moved to v4l2_ext_format as suggested
- v4l2_timecode is still not present in v4l2_ext_buffer, but can be added back thanks to the extra reserved space
- the ENUMFMT is left as is for now, because I think we want Maxime's work on DRM/V4L format unification to land before reworking the ioctl (exposing extra info about the format and not only the 4CC code?). That also means that there's currently no way to know which modifiers are supported
- EXT_FMT/EXT_BUF capability flags to detect whether new ioctls are supported or not have not been added yet

Nothing has changed in v3, just rebased patches on top of media/master so we can discuss it during the Media Summit.

Regards,

Boris

[1]https://www.mail-archive.com/[email protected]/msg135729.html [2]https://github.com/bbrezillon/v4l-utils/commits/master [3]https://github.com/bbrezillon/linux/commit/4882435f80b05a61827649d55cc0f0cee79680a7 [4]https://github.com/bbrezillon/linux/commit/a415216c6aaab2d51f0bd62270b994c8196ddd90

Boris Brezillon (5):
media: v4l2: Extend pixel formats to unify single/multi-planar
handling (and more)
media: videobuf2: Expose helpers to implement the _ext_fmt and
_ext_buf hooks
media: mediabus: Add an helper to convert a ext_pix format to an
mbus_fmt
media: vivid: Convert the capture and output drivers to
EXT_FMT/EXT_BUF
media: vimc: Implement the ext_fmt and ext_buf hooks

Hans Verkuil (1):
media: v4l2: Add extended buffer operations

 .../media/common/videobuf2/videobuf2-core.c   |    2 +
 .../media/common/videobuf2/videobuf2-v4l2.c   |  534 ++++----
 drivers/media/platform/vimc/vimc-capture.c    |   65 +-
 drivers/media/platform/vimc/vimc-common.c     |    4 +-
 drivers/media/platform/vimc/vimc-common.h     |    2 +-
 drivers/media/platform/vivid/vivid-core.c     |   30 +-
 drivers/media/platform/vivid/vivid-vid-cap.c  |  171 +--
 drivers/media/platform/vivid/vivid-vid-cap.h  |   15 +-
 drivers/media/platform/vivid/vivid-vid-out.c  |  195 +--
 drivers/media/platform/vivid/vivid-vid-out.h  |   15 +-
 drivers/media/v4l2-core/v4l2-dev.c            |   54 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          | 1127 +++++++++++++++--
 include/media/v4l2-ioctl.h                    |   63 +
 include/media/v4l2-mediabus.h                 |   22 +
 include/media/videobuf2-core.h                |    6 +-
 include/media/videobuf2-v4l2.h                |   26 +-
 include/uapi/linux/videodev2.h                |  211 +++
 17 files changed, 1897 insertions(+), 645 deletions(-)
Signature · 1 line
2.21.0
#1 lore raw

RFC PATCH v3 1/6 media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more)

This is part of the multiplanar and singleplanar unification process. v4l2_ext_pix_format is supposed to work for both cases.

We also add the concept of modifiers already employed in DRM to expose HW-specific formats (like tiled or compressed formats) and allow exchanging this information with the DRM subsystem in a consistent way.

Note that V4L2_BUF_TYPE_VIDEO[_OUTPUT]_OVERLAY and V4L2_BUF_TYPE_VIDEO_{CAPTURE,OUTPUT}_MPLANE types are no longer accepted in v4l2_ext_format and will be rejected if you use the {G,S,TRY}EXT_FMT ioctls. V4L2_BUF_TYPE_VIDEO_{CAPTURE,OUTPUT}_MPLANE is dropped as part of the multiplanar/singleplanar unification. V4L2_BUF_TYPE_VIDEO[_OUTPUT]_OVERLAY seems to be used mostly on old drivers and supporting it would require some extra rework.

New hooks have been added to v4l2_ioctl_ops to support those new ioctls in drivers, but, in the meantime, the core takes care of converting {S,G,TRY}_EXT_FMT requests into {S,G,TRY}_FMT so that old drivers can still work if the userspace app/lib uses the new ioctls.
The conversion is also done the other around to allow userspace apps/libs using {S,G,TRY}_FMT to work with drivers implementing the _ext_ hooks.

---
Changes in v3:
- Rebased on top of media/master (post 5.4-rc1)

Changes in v2:
- Move the modifier in v4l2_ext_format (was formerly placed in v4l2_ext_plane)
- Fix a few bugs in the converters and add a strict parameter to allow conversion of uninitialized/mis-initialized objects

---
 drivers/media/v4l2-core/v4l2-dev.c   |  24 +-
 drivers/media/v4l2-core/v4l2-ioctl.c | 699 ++++++++++++++++++++++++---
 include/media/v4l2-ioctl.h           |  33 ++
 include/uapi/linux/videodev2.h       |  81 ++++
 4 files changed, 754 insertions(+), 83 deletions(-)
drivers/media/v4l2-core/v4l2-dev.c +18 -6
index 4037689a945a..9aad715537b3 100644
@@ -595,32 +595,44 @@ static void determine_valid_ioctls(struct video_device *vdev)
 			       ops->vidioc_enum_fmt_meta_out)))
 			set_bit(_IOC_NR(VIDIOC_ENUM_FMT), valid_ioctls);
 		if ((is_rx && (ops->vidioc_g_fmt_vid_cap ||
+			       ops->vidioc_g_ext_fmt_vid_cap ||
 			       ops->vidioc_g_fmt_vid_cap_mplane ||
 			       ops->vidioc_g_fmt_vid_overlay ||
 			       ops->vidioc_g_fmt_meta_cap)) ||
 		    (is_tx && (ops->vidioc_g_fmt_vid_out ||
+			       ops->vidioc_g_ext_fmt_vid_out ||
 			       ops->vidioc_g_fmt_vid_out_mplane ||
 			       ops->vidioc_g_fmt_vid_out_overlay ||
-			       ops->vidioc_g_fmt_meta_out)))
-			 set_bit(_IOC_NR(VIDIOC_G_FMT), valid_ioctls);
+			       ops->vidioc_g_fmt_meta_out))) {
+			set_bit(_IOC_NR(VIDIOC_G_FMT), valid_ioctls);
+			set_bit(_IOC_NR(VIDIOC_G_EXT_FMT), valid_ioctls);
+		}
 		if ((is_rx && (ops->vidioc_s_fmt_vid_cap ||
+			       ops->vidioc_s_ext_fmt_vid_cap ||
 			       ops->vidioc_s_fmt_vid_cap_mplane ||
 			       ops->vidioc_s_fmt_vid_overlay ||
 			       ops->vidioc_s_fmt_meta_cap)) ||
 		    (is_tx && (ops->vidioc_s_fmt_vid_out ||
+			       ops->vidioc_s_ext_fmt_vid_out ||
 			       ops->vidioc_s_fmt_vid_out_mplane ||
 			       ops->vidioc_s_fmt_vid_out_overlay ||
-			       ops->vidioc_s_fmt_meta_out)))
-			 set_bit(_IOC_NR(VIDIOC_S_FMT), valid_ioctls);
+			       ops->vidioc_s_fmt_meta_out))) {
+			set_bit(_IOC_NR(VIDIOC_S_FMT), valid_ioctls);
+			set_bit(_IOC_NR(VIDIOC_S_EXT_FMT), valid_ioctls);
+		}
 		if ((is_rx && (ops->vidioc_try_fmt_vid_cap ||
+			       ops->vidioc_try_ext_fmt_vid_cap ||
 			       ops->vidioc_try_fmt_vid_cap_mplane ||
 			       ops->vidioc_try_fmt_vid_overlay ||
 			       ops->vidioc_try_fmt_meta_cap)) ||
 		    (is_tx && (ops->vidioc_try_fmt_vid_out ||
+			       ops->vidioc_try_ext_fmt_vid_out ||
 			       ops->vidioc_try_fmt_vid_out_mplane ||
 			       ops->vidioc_try_fmt_vid_out_overlay ||
-			       ops->vidioc_try_fmt_meta_out)))
-			 set_bit(_IOC_NR(VIDIOC_TRY_FMT), valid_ioctls);
+			       ops->vidioc_try_fmt_meta_out))) {
+			set_bit(_IOC_NR(VIDIOC_TRY_FMT), valid_ioctls);
+			set_bit(_IOC_NR(VIDIOC_TRY_EXT_FMT), valid_ioctls);
+		}
 		SET_VALID_IOCTL(ops, VIDIOC_OVERLAY, vidioc_overlay);
 		SET_VALID_IOCTL(ops, VIDIOC_G_FBUF, vidioc_g_fbuf);
 		SET_VALID_IOCTL(ops, VIDIOC_S_FBUF, vidioc_s_fbuf);
drivers/media/v4l2-core/v4l2-ioctl.c +622 -77

817 lines not on this page · raw message

include/media/v4l2-ioctl.h +33 -0
index 4bba65a59d46..39ac07fbc7b7 100644
@@ -48,11 +48,16 @@ struct v4l2_fh;
  * @vidioc_g_fmt_vid_cap: pointer to the function that implements
  *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video capture
  *	in single plane mode
+ * @vidioc_g_ext_fmt_vid_cap: pointer to the function that implements
+ *	:ref:`VIDIOC_G_EXT_FMT <vidioc_g_ext_fmt>` ioctl logic for video
+ *	capture
  * @vidioc_g_fmt_vid_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay
  * @vidioc_g_fmt_vid_out: pointer to the function that implements
  *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video out
  *	in single plane mode
+ * @vidioc_g_ext_fmt_vid_out: pointer to the function that implements
+ *	:ref:`VIDIOC_G_EXT_FMT <vidioc_g_ext_fmt>` ioctl logic for video out
  * @vidioc_g_fmt_vid_out_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
  * @vidioc_g_fmt_vbi_cap: pointer to the function that implements
@@ -82,11 +87,16 @@ struct v4l2_fh;
  * @vidioc_s_fmt_vid_cap: pointer to the function that implements
  *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video capture
  *	in single plane mode
+ * @vidioc_s_ext_fmt_vid_cap: pointer to the function that implements
+ *	:ref:`VIDIOC_S_EXT_FMT <vidioc_s_ext_fmt>` ioctl logic for video
+ *	capture
  * @vidioc_s_fmt_vid_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay
  * @vidioc_s_fmt_vid_out: pointer to the function that implements
  *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video out
  *	in single plane mode
+ * @vidioc_s_ext_fmt_vid_out: pointer to the function that implements
+ *	:ref:`VIDIOC_S_EXT_FMT <vidioc_g_fmt>` ioctl logic for video out
  * @vidioc_s_fmt_vid_out_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
  * @vidioc_s_fmt_vbi_cap: pointer to the function that implements
@@ -116,11 +126,16 @@ struct v4l2_fh;
  * @vidioc_try_fmt_vid_cap: pointer to the function that implements
  *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video capture
  *	in single plane mode
+ * @vidioc_try_ext_fmt_vid_cap: pointer to the function that implements
+ *	:ref:`VIDIOC_TRY_EXT_FMT <vidioc_try_ext_fmt>` ioctl logic for video
+	capture
  * @vidioc_try_fmt_vid_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
  * @vidioc_try_fmt_vid_out: pointer to the function that implements
  *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video out
  *	in single plane mode
+ * @vidioc_try_ext_fmt_vid_out: pointer to the function that implements
+ *	:ref:`VIDIOC_TRY_EXT_FMT <vidioc_g_fmt>` ioctl logic for video out
  * @vidioc_try_fmt_vid_out_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
  *	output
@@ -319,10 +334,14 @@ struct v4l2_ioctl_ops {
 	/* VIDIOC_G_FMT handlers */
 	int (*vidioc_g_fmt_vid_cap)(struct file *file, void *fh,
 				    struct v4l2_format *f);
+	int (*vidioc_g_ext_fmt_vid_cap)(struct file *file, void *fh,
+					struct v4l2_ext_pix_format *f);
 	int (*vidioc_g_fmt_vid_overlay)(struct file *file, void *fh,
 					struct v4l2_format *f);
 	int (*vidioc_g_fmt_vid_out)(struct file *file, void *fh,
 				    struct v4l2_format *f);
+	int (*vidioc_g_ext_fmt_vid_out)(struct file *file, void *fh,
+					struct v4l2_ext_pix_format *f);
 	int (*vidioc_g_fmt_vid_out_overlay)(struct file *file, void *fh,
 					    struct v4l2_format *f);
 	int (*vidioc_g_fmt_vbi_cap)(struct file *file, void *fh,
@@ -349,10 +368,14 @@ struct v4l2_ioctl_ops {
 	/* VIDIOC_S_FMT handlers */
 	int (*vidioc_s_fmt_vid_cap)(struct file *file, void *fh,
 				    struct v4l2_format *f);
+	int (*vidioc_s_ext_fmt_vid_cap)(struct file *file, void *fh,
+					struct v4l2_ext_pix_format *f);
 	int (*vidioc_s_fmt_vid_overlay)(struct file *file, void *fh,
 					struct v4l2_format *f);
 	int (*vidioc_s_fmt_vid_out)(struct file *file, void *fh,
 				    struct v4l2_format *f);
+	int (*vidioc_s_ext_fmt_vid_out)(struct file *file, void *fh,
+					struct v4l2_ext_pix_format *f);
 	int (*vidioc_s_fmt_vid_out_overlay)(struct file *file, void *fh,
 					    struct v4l2_format *f);
 	int (*vidioc_s_fmt_vbi_cap)(struct file *file, void *fh,
@@ -379,10 +402,14 @@ struct v4l2_ioctl_ops {
 	/* VIDIOC_TRY_FMT handlers */
 	int (*vidioc_try_fmt_vid_cap)(struct file *file, void *fh,
 				      struct v4l2_format *f);
+	int (*vidioc_try_ext_fmt_vid_cap)(struct file *file, void *fh,
+					  struct v4l2_ext_pix_format *f);
 	int (*vidioc_try_fmt_vid_overlay)(struct file *file, void *fh,
 					  struct v4l2_format *f);
 	int (*vidioc_try_fmt_vid_out)(struct file *file, void *fh,
 				      struct v4l2_format *f);
+	int (*vidioc_try_ext_fmt_vid_out)(struct file *file, void *fh,
+					  struct v4l2_ext_pix_format *f);
 	int (*vidioc_try_fmt_vid_out_overlay)(struct file *file, void *fh,
 					     struct v4l2_format *f);
 	int (*vidioc_try_fmt_vbi_cap)(struct file *file, void *fh,
@@ -724,4 +751,10 @@ long int video_usercopy(struct file *file, unsigned int cmd,
 long int video_ioctl2(struct file *file,
 		      unsigned int cmd, unsigned long int arg);
 
+int v4l2_format_to_ext_format(const struct v4l2_format *f,
+			      struct v4l2_ext_format *e, bool strict);
+int v4l2_ext_format_to_format(const struct v4l2_ext_format *e,
+			      struct v4l2_format *f,
+			      bool mplane_cap, bool strict);
+
 #endif /* _V4L2_IOCTL_H */
include/uapi/linux/videodev2.h +81 -0

99 lines not on this page · raw message

Signature · 1 line
2.21.0
#2 lore raw

Re: [RFC PATCH v3 1/6] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more)

Quoted 25 lines · Boris Brezillon

This is part of the multiplanar and singleplanar unification process. v4l2_ext_pix_format is supposed to work for both cases.

We also add the concept of modifiers already employed in DRM to expose HW-specific formats (like tiled or compressed formats) and allow exchanging this information with the DRM subsystem in a consistent way.

Note that V4L2_BUF_TYPE_VIDEO[_OUTPUT]_OVERLAY and V4L2_BUF_TYPE_VIDEO_{CAPTURE,OUTPUT}_MPLANE types are no longer accepted in v4l2_ext_format and will be rejected if you use the {G,S,TRY}EXT_FMT ioctls. V4L2_BUF_TYPE_VIDEO_{CAPTURE,OUTPUT}_MPLANE is dropped as part of the multiplanar/singleplanar unification. V4L2_BUF_TYPE_VIDEO[_OUTPUT]_OVERLAY seems to be used mostly on old drivers and supporting it would require some extra rework.

New hooks have been added to v4l2_ioctl_ops to support those new ioctls in drivers, but, in the meantime, the core takes care of converting {S,G,TRY}_EXT_FMT requests into {S,G,TRY}_FMT so that old drivers can still work if the userspace app/lib uses the new ioctls.
The conversion is also done the other around to allow userspace apps/libs using {S,G,TRY}_FMT to work with drivers implementing the _ext_ hooks.

---

<snip>

include/uapi/linux/videodev2.h struct v4l2_create_buffers { · line 2590

Quoted 8 lines
include/uapi/linux/videodev2.h +3 -0
+#define VIDIOC_G_EXT_FMT	_IOWR('V', 104, struct v4l2_ext_format)
+#define VIDIOC_S_EXT_FMT	_IOWR('V', 105, struct v4l2_ext_format)
+#define VIDIOC_TRY_EXT_FMT	_IOWR('V', 106, struct v4l2_ext_format)
 /* Reminder: when adding new ioctls please add support for them to
    drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */

Since we're extending g/s/try_fmt, we should also provide a replacement for enum_fmt, esp. given this thread:

https://www.mail-archive.com/[email protected]/msg150871.html

So here is a preliminary suggestion:

struct v4l2_ext_fmtdesc {

        __u32               index;             /* Format number      */
        __u32               type;              /* enum v4l2_buf_type */

__u32 which; /* enum v4l2_subdev_format_whence, ignored if mbus_code == 0 */
__u32 mbus_code; /* Mediabus Code (set to 0 if n/a) */
__u32 flags;
__u8 description[32]; /* Description string */
__u32 pixelformat; /* Format fourcc */
};

This would solve (I think) the issue raised in the thread since you can now get just for formats that are valid for the given mediabus code and the which field.

Other improvements that could be made is to return more information about the format (similar to struct v4l2_format_info). In particular v4l2_pixel_encoding and mem/comp_planes would be useful for userspace to know.

Finally, we can also add a new ioctl that combines ENUM_FMT/ENUM_FRAMESIZES/ENUM_FRAMEINTERVALS and returns an array of all valid formats/sizes/intervals, requiring just a single ioctl to obtain all this information.

Regards,

Hans

maintainer
#3 lore raw

Re: [RFC PATCH v3 1/6] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more)

Quoted 42 lines · Hans Verkuil
Quoted 25 lines · Boris Brezillon

This is part of the multiplanar and singleplanar unification process. v4l2_ext_pix_format is supposed to work for both cases.

We also add the concept of modifiers already employed in DRM to expose HW-specific formats (like tiled or compressed formats) and allow exchanging this information with the DRM subsystem in a consistent way.

Note that V4L2_BUF_TYPE_VIDEO[_OUTPUT]_OVERLAY and V4L2_BUF_TYPE_VIDEO_{CAPTURE,OUTPUT}_MPLANE types are no longer accepted in v4l2_ext_format and will be rejected if you use the {G,S,TRY}EXT_FMT ioctls. V4L2_BUF_TYPE_VIDEO_{CAPTURE,OUTPUT}_MPLANE is dropped as part of the multiplanar/singleplanar unification. V4L2_BUF_TYPE_VIDEO[_OUTPUT]_OVERLAY seems to be used mostly on old drivers and supporting it would require some extra rework.

New hooks have been added to v4l2_ioctl_ops to support those new ioctls in drivers, but, in the meantime, the core takes care of converting {S,G,TRY}_EXT_FMT requests into {S,G,TRY}_FMT so that old drivers can still work if the userspace app/lib uses the new ioctls.
The conversion is also done the other around to allow userspace apps/libs using {S,G,TRY}_FMT to work with drivers implementing the _ext_ hooks.

---

<snip>

include/uapi/linux/videodev2.h struct v4l2_create_buffers { · line 2590

Quoted 8 lines
include/uapi/linux/videodev2.h +3 -0
+#define VIDIOC_G_EXT_FMT     _IOWR('V', 104, struct v4l2_ext_format)
+#define VIDIOC_S_EXT_FMT     _IOWR('V', 105, struct v4l2_ext_format)
+#define VIDIOC_TRY_EXT_FMT   _IOWR('V', 106, struct v4l2_ext_format)
 /* Reminder: when adding new ioctls please add support for them to
    drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */

Since we're extending g/s/try_fmt, we should also provide a replacement for enum_fmt, esp. given this thread:

Quoted 1 line

https://www.mail-archive.com/[email protected]/msg150871.html

While that's a completely valid problem that should be addressed, I'm not sure if putting all the things in one bag would have a positive effect on solving all the problems in a reasonable timeline.

Quoted 11 lines

So here is a preliminary suggestion:

struct v4l2_ext_fmtdesc {

        __u32               index;             /* Format number      */
        __u32               type;              /* enum v4l2_buf_type */
        __u32               which;             /* enum v4l2_subdev_format_whence, ignored if mbus_code == 0 */
        __u32               mbus_code;         /* Mediabus Code (set to 0 if n/a) */
        __u32               flags;
        __u8                description[32];   /* Description string */
        __u32               pixelformat;       /* Format fourcc      */
Quoted 5 lines

};

This would solve (I think) the issue raised in the thread since you can now get just for formats that are valid for the given mediabus code and the which field.

Hmm, why only mbus_code? We have the same problem with mem2mem devices, where the format set on one queue affects the formats supported on another queue. Perhaps it should be defined to return formats valid with the current state of the driver? If we want to extend it to return formats for arbitrary states, perhaps we should use a mechanism similar to the TRY_FMT slot in subdevices, where we can set the configuration we want to test against and then ENUM_FMT would return the formats valid for that configuration?

Quoted 3 lines

Other improvements that could be made is to return more information about the format (similar to struct v4l2_format_info). In particular v4l2_pixel_encoding and mem/comp_planes would be useful for userspace to know.

An alternative would be to go away from mixing mem_planes and pixelformats and just having the "planarity" queryable and configurable separately. The existing duplicated FourCCs would have to remain for compatibility reasons, though.

Quoted 4 lines

Finally, we can also add a new ioctl that combines ENUM_FMT/ENUM_FRAMESIZES/ENUM_FRAMEINTERVALS and returns an array of all valid formats/sizes/intervals, requiring just a single ioctl to obtain all this information.

While it definitely sounds like a useful thing to have, it would be an interesting problem to solve given the inter-dependencies between pads, queues and other state, as in the mbus example above.

Best regards,
Tomasz

#4 lore raw

Re: [RFC PATCH v3 1/6] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more)

Quoted 43 lines · Tomasz Figa
Quoted 42 lines · Hans Verkuil
Quoted 25 lines · Boris Brezillon

This is part of the multiplanar and singleplanar unification process. v4l2_ext_pix_format is supposed to work for both cases.

We also add the concept of modifiers already employed in DRM to expose HW-specific formats (like tiled or compressed formats) and allow exchanging this information with the DRM subsystem in a consistent way.

Note that V4L2_BUF_TYPE_VIDEO[_OUTPUT]_OVERLAY and V4L2_BUF_TYPE_VIDEO_{CAPTURE,OUTPUT}_MPLANE types are no longer accepted in v4l2_ext_format and will be rejected if you use the {G,S,TRY}EXT_FMT ioctls. V4L2_BUF_TYPE_VIDEO_{CAPTURE,OUTPUT}_MPLANE is dropped as part of the multiplanar/singleplanar unification. V4L2_BUF_TYPE_VIDEO[_OUTPUT]_OVERLAY seems to be used mostly on old drivers and supporting it would require some extra rework.

New hooks have been added to v4l2_ioctl_ops to support those new ioctls in drivers, but, in the meantime, the core takes care of converting {S,G,TRY}_EXT_FMT requests into {S,G,TRY}_FMT so that old drivers can still work if the userspace app/lib uses the new ioctls.
The conversion is also done the other around to allow userspace apps/libs using {S,G,TRY}_FMT to work with drivers implementing the _ext_ hooks.

---

<snip>

include/uapi/linux/videodev2.h struct v4l2_create_buffers { · line 2590

Quoted 8 lines
include/uapi/linux/videodev2.h +3 -0
+#define VIDIOC_G_EXT_FMT     _IOWR('V', 104, struct v4l2_ext_format)
+#define VIDIOC_S_EXT_FMT     _IOWR('V', 105, struct v4l2_ext_format)
+#define VIDIOC_TRY_EXT_FMT   _IOWR('V', 106, struct v4l2_ext_format)
 /* Reminder: when adding new ioctls please add support for them to
    drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */

Since we're extending g/s/try_fmt, we should also provide a replacement for enum_fmt, esp. given this thread:

Quoted 5 lines
Quoted 1 line

https://www.mail-archive.com/[email protected]/msg150871.html

While that's a completely valid problem that should be addressed, I'm not sure if putting all the things in one bag would have a positive effect on solving all the problems in a reasonable timeline.

I'm not sure what you mean with this. We can't ignore this either, and if we're going to add these new ioctls, then let's try to fix as much as we can.

Quoted 18 lines
Quoted 12 lines

So here is a preliminary suggestion:

struct v4l2_ext_fmtdesc {

        __u32               index;             /* Format number      */
        __u32               type;              /* enum v4l2_buf_type */
        __u32               which;             /* enum v4l2_subdev_format_whence, ignored if mbus_code == 0 */
        __u32               mbus_code;         /* Mediabus Code (set to 0 if n/a) */
        __u32               flags;
        __u8                description[32];   /* Description string */
        __u32               pixelformat;       /* Format fourcc      */
Quoted 5 lines

};

This would solve (I think) the issue raised in the thread since you can now get just for formats that are valid for the given mediabus code and the which field.

Quoted 8 lines

Hmm, why only mbus_code? We have the same problem with mem2mem devices, where the format set on one queue affects the formats supported on another queue. Perhaps it should be defined to return formats valid with the current state of the driver? If we want to extend it to return formats for arbitrary states, perhaps we should use a mechanism similar to the TRY_FMT slot in subdevices, where we can set the configuration we want to test against and then ENUM_FMT would return the formats valid for that configuration?

Good point.

Quoted 9 lines
Quoted 4 lines

Other improvements that could be made is to return more information about the format (similar to struct v4l2_format_info). In particular v4l2_pixel_encoding and mem/comp_planes would be useful for userspace to know.

An alternative would be to go away from mixing mem_planes and pixelformats and just having the "planarity" queryable and configurable separately. The existing duplicated FourCCs would have to remain for compatibility reasons, though.

Interesting idea, but I don't know if this would make the API more or less confusing.

Quoted 9 lines
Quoted 5 lines

Finally, we can also add a new ioctl that combines ENUM_FMT/ENUM_FRAMESIZES/ENUM_FRAMEINTERVALS and returns an array of all valid formats/sizes/intervals, requiring just a single ioctl to obtain all this information.

While it definitely sounds like a useful thing to have, it would be an interesting problem to solve given the inter-dependencies between pads, queues and other state, as in the mbus example above.

This is definitely a separate issue for further in the future.

Regards,

Hans

maintainer
#5 lore raw

Re: [RFC PATCH v3 1/6] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more)

Quoted 45 lines · Hans Verkuil
Quoted 43 lines · Tomasz Figa
Quoted 42 lines · Hans Verkuil
Quoted 25 lines · Boris Brezillon

This is part of the multiplanar and singleplanar unification process. v4l2_ext_pix_format is supposed to work for both cases.

We also add the concept of modifiers already employed in DRM to expose HW-specific formats (like tiled or compressed formats) and allow exchanging this information with the DRM subsystem in a consistent way.

Note that V4L2_BUF_TYPE_VIDEO[_OUTPUT]_OVERLAY and V4L2_BUF_TYPE_VIDEO_{CAPTURE,OUTPUT}_MPLANE types are no longer accepted in v4l2_ext_format and will be rejected if you use the {G,S,TRY}EXT_FMT ioctls. V4L2_BUF_TYPE_VIDEO_{CAPTURE,OUTPUT}_MPLANE is dropped as part of the multiplanar/singleplanar unification. V4L2_BUF_TYPE_VIDEO[_OUTPUT]_OVERLAY seems to be used mostly on old drivers and supporting it would require some extra rework.

New hooks have been added to v4l2_ioctl_ops to support those new ioctls in drivers, but, in the meantime, the core takes care of converting {S,G,TRY}_EXT_FMT requests into {S,G,TRY}_FMT so that old drivers can still work if the userspace app/lib uses the new ioctls.
The conversion is also done the other around to allow userspace apps/libs using {S,G,TRY}_FMT to work with drivers implementing the _ext_ hooks.

---

<snip>

include/uapi/linux/videodev2.h struct v4l2_create_buffers { · line 2590

Quoted 8 lines
include/uapi/linux/videodev2.h +3 -0
+#define VIDIOC_G_EXT_FMT     _IOWR('V', 104, struct v4l2_ext_format)
+#define VIDIOC_S_EXT_FMT     _IOWR('V', 105, struct v4l2_ext_format)
+#define VIDIOC_TRY_EXT_FMT   _IOWR('V', 106, struct v4l2_ext_format)
 /* Reminder: when adding new ioctls please add support for them to
    drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */

Since we're extending g/s/try_fmt, we should also provide a replacement for enum_fmt, esp. given this thread:

Quoted 9 lines
Quoted 5 lines
Quoted 1 line

https://www.mail-archive.com/[email protected]/msg150871.html

While that's a completely valid problem that should be addressed, I'm not sure if putting all the things in one bag would have a positive effect on solving all the problems in a reasonable timeline.

I'm not sure what you mean with this. We can't ignore this either, and if we're going to add these new ioctls, then let's try to fix as much as we can.

My point is that this series solves a completely orthogonal problem without a need to touch ENUM_FMT at all. Is there any reason why solving this particular problem separately would make solving the ENUM_FMT problem more difficult in the future?

Quoted 27 lines
Quoted 18 lines
Quoted 12 lines

So here is a preliminary suggestion:

struct v4l2_ext_fmtdesc {

        __u32               index;             /* Format number      */
        __u32               type;              /* enum v4l2_buf_type */
        __u32               which;             /* enum v4l2_subdev_format_whence, ignored if mbus_code == 0 */
        __u32               mbus_code;         /* Mediabus Code (set to 0 if n/a) */
        __u32               flags;
        __u8                description[32];   /* Description string */
        __u32               pixelformat;       /* Format fourcc      */
Quoted 5 lines

};

This would solve (I think) the issue raised in the thread since you can now get just for formats that are valid for the given mediabus code and the which field.

Quoted 8 lines

Hmm, why only mbus_code? We have the same problem with mem2mem devices, where the format set on one queue affects the formats supported on another queue. Perhaps it should be defined to return formats valid with the current state of the driver? If we want to extend it to return formats for arbitrary states, perhaps we should use a mechanism similar to the TRY_FMT slot in subdevices, where we can set the configuration we want to test against and then ENUM_FMT would return the formats valid for that configuration?

Quoted 2 lines

Good point.

We might want to keep the control'ification of the API in mind, which should simplify dealing with state inter-dependencies, because all the state would be represented in a uniform way.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

Other improvements that could be made is to return more information about the format (similar to struct v4l2_format_info). In particular v4l2_pixel_encoding and mem/comp_planes would be useful for userspace to know.

Quoted 7 lines
Quoted 4 lines

An alternative would be to go away from mixing mem_planes and pixelformats and just having the "planarity" queryable and configurable separately. The existing duplicated FourCCs would have to remain for compatibility reasons, though.

Interesting idea, but I don't know if this would make the API more or less confusing.

Yeah, this definitely needs more thought. My experience with working with many people trying to use V4L2 in the userspace tells me that the existing model is confusing, though. DRM and most userspace libraries use FourCCs only to define the pixelformats themselves and planarity is a separate aspect.

The target model that I'd see happening would be to have the multiple-memory plane semantics used everywhere, so all color planes are described as separate entities, up to having different DMA-buf FDs. Then the single memory plane case would be just one of the specific cases, where all the DMA-buf FDs point to the same buffer and color plane offsets are laid out contiguously, which could be enforced by generic code when queuing the buffer if that's a hardware requirement.

Quoted 11 lines
Quoted 9 lines
Quoted 5 lines

Finally, we can also add a new ioctl that combines ENUM_FMT/ENUM_FRAMESIZES/ENUM_FRAMEINTERVALS and returns an array of all valid formats/sizes/intervals, requiring just a single ioctl to obtain all this information.

While it definitely sounds like a useful thing to have, it would be an interesting problem to solve given the inter-dependencies between pads, queues and other state, as in the mbus example above.

This is definitely a separate issue for further in the future.

Agreed.

Best regards,
Tomasz

#6 lore raw

Re: [RFC PATCH v3 1/6] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more)

Quoted 52 lines · Tomasz Figa
Quoted 51 lines · Hans Verkuil
Quoted 48 lines · Tomasz Figa
Quoted 42 lines · Hans Verkuil
Quoted 25 lines · Boris Brezillon

This is part of the multiplanar and singleplanar unification process. v4l2_ext_pix_format is supposed to work for both cases.

We also add the concept of modifiers already employed in DRM to expose HW-specific formats (like tiled or compressed formats) and allow exchanging this information with the DRM subsystem in a consistent way.

Note that V4L2_BUF_TYPE_VIDEO[_OUTPUT]_OVERLAY and V4L2_BUF_TYPE_VIDEO_{CAPTURE,OUTPUT}_MPLANE types are no longer accepted in v4l2_ext_format and will be rejected if you use the {G,S,TRY}EXT_FMT ioctls. V4L2_BUF_TYPE_VIDEO_{CAPTURE,OUTPUT}_MPLANE is dropped as part of the multiplanar/singleplanar unification. V4L2_BUF_TYPE_VIDEO[_OUTPUT]_OVERLAY seems to be used mostly on old drivers and supporting it would require some extra rework.

New hooks have been added to v4l2_ioctl_ops to support those new ioctls in drivers, but, in the meantime, the core takes care of converting {S,G,TRY}_EXT_FMT requests into {S,G,TRY}_FMT so that old drivers can still work if the userspace app/lib uses the new ioctls.
The conversion is also done the other around to allow userspace apps/libs using {S,G,TRY}_FMT to work with drivers implementing the _ext_ hooks.

---

<snip>

include/uapi/linux/videodev2.h struct v4l2_create_buffers { · line 2590

Quoted 8 lines
include/uapi/linux/videodev2.h +3 -0
+#define VIDIOC_G_EXT_FMT     _IOWR('V', 104, struct v4l2_ext_format)
+#define VIDIOC_S_EXT_FMT     _IOWR('V', 105, struct v4l2_ext_format)
+#define VIDIOC_TRY_EXT_FMT   _IOWR('V', 106, struct v4l2_ext_format)
 /* Reminder: when adding new ioctls please add support for them to
    drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */

Since we're extending g/s/try_fmt, we should also provide a replacement for enum_fmt, esp. given this thread:

Quoted 1 line

https://www.mail-archive.com/[email protected]/msg150871.html

While that's a completely valid problem that should be addressed, I'm not sure if putting all the things in one bag would have a positive effect on solving all the problems in a reasonable timeline.

Quoted 8 lines
Quoted 3 lines

I'm not sure what you mean with this. We can't ignore this either, and if we're going to add these new ioctls, then let's try to fix as much as we can.

My point is that this series solves a completely orthogonal problem without a need to touch ENUM_FMT at all. Is there any reason why solving this particular problem separately would make solving the ENUM_FMT problem more difficult in the future?

I do not agree with you that this is an orthogonal problem. If we are creating new FMT ioctls to solve various problems, then it makes sense to look at ALL the problems, including this. We might decide to postpone creating a EXT_ENUM_FMT ioctl, but during the discussion we should look at this issue as well.

To take one suggestion you made: use of the active/try slots for existing non-MC drivers. If we decide to go into that direction (and I think it is a very interesting idea), then that requires that ENUM_FMT is taken into account anyway. And probably other ioctls such as the selection API as well.

I think we need to think big here, and try to at least explore the possibility of creating an API that tries to limit the differences between video and subdev nodes.

Quoted 28 lines
Quoted 28 lines
Quoted 27 lines
Quoted 13 lines

So here is a preliminary suggestion:

struct v4l2_ext_fmtdesc {

        __u32               index;             /* Format number      */
        __u32               type;              /* enum v4l2_buf_type */
        __u32               which;             /* enum v4l2_subdev_format_whence, ignored if mbus_code == 0 */
        __u32               mbus_code;         /* Mediabus Code (set to 0 if n/a) */
        __u32               flags;
        __u8                description[32];   /* Description string */
        __u32               pixelformat;       /* Format fourcc      */
Quoted 5 lines

};

This would solve (I think) the issue raised in the thread since you can now get just for formats that are valid for the given mediabus code and the which field.

Hmm, why only mbus_code? We have the same problem with mem2mem devices, where the format set on one queue affects the formats supported on another queue. Perhaps it should be defined to return formats valid with the current state of the driver? If we want to extend it to return formats for arbitrary states, perhaps we should use a mechanism similar to the TRY_FMT slot in subdevices, where we can set the configuration we want to test against and then ENUM_FMT would return the formats valid for that configuration?

Quoted 6 lines
Quoted 2 lines

Good point.

We might want to keep the control'ification of the API in mind, which should simplify dealing with state inter-dependencies, because all the state would be represented in a uniform way.

I still don't know what Laurent wants to do with that.

Quoted 20 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

Other improvements that could be made is to return more information about the format (similar to struct v4l2_format_info). In particular v4l2_pixel_encoding and mem/comp_planes would be useful for userspace to know.

Quoted 7 lines
Quoted 4 lines

An alternative would be to go away from mixing mem_planes and pixelformats and just having the "planarity" queryable and configurable separately. The existing duplicated FourCCs would have to remain for compatibility reasons, though.

Interesting idea, but I don't know if this would make the API more or less confusing.

Yeah, this definitely needs more thought. My experience with working with many people trying to use V4L2 in the userspace tells me that the existing model is confusing, though. DRM and most userspace libraries use FourCCs only to define the pixelformats themselves and planarity is a separate aspect.

Quoted 8 lines

The target model that I'd see happening would be to have the multiple-memory plane semantics used everywhere, so all color planes are described as separate entities, up to having different DMA-buf FDs. Then the single memory plane case would be just one of the specific cases, where all the DMA-buf FDs point to the same buffer and color plane offsets are laid out contiguously, which could be enforced by generic code when queuing the buffer if that's a hardware requirement.

Do you think you can come up with a rough proposal before the ELCE session?

Regards,

Hans

Quoted 18 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

Finally, we can also add a new ioctl that combines ENUM_FMT/ENUM_FRAMESIZES/ENUM_FRAMEINTERVALS and returns an array of all valid formats/sizes/intervals, requiring just a single ioctl to obtain all this information.

Quoted 5 lines
Quoted 3 lines

While it definitely sounds like a useful thing to have, it would be an interesting problem to solve given the inter-dependencies between pads, queues and other state, as in the mbus example above.

This is definitely a separate issue for further in the future.

Agreed.

Best regards,
Tomasz

maintainer
#7 lore raw

Re: [RFC PATCH v3 1/6] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more)

Quoted 68 lines · Hans Verkuil
Quoted 52 lines · Tomasz Figa
Quoted 51 lines · Hans Verkuil
Quoted 48 lines · Tomasz Figa
Quoted 42 lines · Hans Verkuil
Quoted 25 lines · Boris Brezillon

This is part of the multiplanar and singleplanar unification process. v4l2_ext_pix_format is supposed to work for both cases.

We also add the concept of modifiers already employed in DRM to expose HW-specific formats (like tiled or compressed formats) and allow exchanging this information with the DRM subsystem in a consistent way.

Note that V4L2_BUF_TYPE_VIDEO[_OUTPUT]_OVERLAY and V4L2_BUF_TYPE_VIDEO_{CAPTURE,OUTPUT}_MPLANE types are no longer accepted in v4l2_ext_format and will be rejected if you use the {G,S,TRY}EXT_FMT ioctls. V4L2_BUF_TYPE_VIDEO_{CAPTURE,OUTPUT}_MPLANE is dropped as part of the multiplanar/singleplanar unification. V4L2_BUF_TYPE_VIDEO[_OUTPUT]_OVERLAY seems to be used mostly on old drivers and supporting it would require some extra rework.

New hooks have been added to v4l2_ioctl_ops to support those new ioctls in drivers, but, in the meantime, the core takes care of converting {S,G,TRY}_EXT_FMT requests into {S,G,TRY}_FMT so that old drivers can still work if the userspace app/lib uses the new ioctls.
The conversion is also done the other around to allow userspace apps/libs using {S,G,TRY}_FMT to work with drivers implementing the _ext_ hooks.

---

<snip>

include/uapi/linux/videodev2.h struct v4l2_create_buffers { · line 2590

Quoted 8 lines
include/uapi/linux/videodev2.h +3 -0
+#define VIDIOC_G_EXT_FMT     _IOWR('V', 104, struct v4l2_ext_format)
+#define VIDIOC_S_EXT_FMT     _IOWR('V', 105, struct v4l2_ext_format)
+#define VIDIOC_TRY_EXT_FMT   _IOWR('V', 106, struct v4l2_ext_format)
 /* Reminder: when adding new ioctls please add support for them to
    drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */

Since we're extending g/s/try_fmt, we should also provide a replacement for enum_fmt, esp. given this thread:

Quoted 1 line

https://www.mail-archive.com/[email protected]/msg150871.html

While that's a completely valid problem that should be addressed, I'm not sure if putting all the things in one bag would have a positive effect on solving all the problems in a reasonable timeline.

Quoted 8 lines
Quoted 3 lines

I'm not sure what you mean with this. We can't ignore this either, and if we're going to add these new ioctls, then let's try to fix as much as we can.

My point is that this series solves a completely orthogonal problem without a need to touch ENUM_FMT at all. Is there any reason why solving this particular problem separately would make solving the ENUM_FMT problem more difficult in the future?

I do not agree with you that this is an orthogonal problem. If we are creating new FMT ioctls to solve various problems, then it makes sense to look at ALL the problems, including this. We might decide to postpone creating a EXT_ENUM_FMT ioctl, but during the discussion we should look at this issue as well.

Quoted 9 lines

To take one suggestion you made: use of the active/try slots for existing non-MC drivers. If we decide to go into that direction (and I think it is a very interesting idea), then that requires that ENUM_FMT is taken into account anyway. And probably other ioctls such as the selection API as well.

I think we need to think big here, and try to at least explore the possibility of creating an API that tries to limit the differences between video and subdev nodes.

Okay, agreed on this.

Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 27 lines
Quoted 13 lines

So here is a preliminary suggestion:

struct v4l2_ext_fmtdesc {

        __u32               index;             /* Format number      */
        __u32               type;              /* enum v4l2_buf_type */
        __u32               which;             /* enum v4l2_subdev_format_whence, ignored if mbus_code == 0 */
        __u32               mbus_code;         /* Mediabus Code (set to 0 if n/a) */
        __u32               flags;
        __u8                description[32];   /* Description string */
        __u32               pixelformat;       /* Format fourcc      */
Quoted 5 lines

};

This would solve (I think) the issue raised in the thread since you can now get just for formats that are valid for the given mediabus code and the which field.

Hmm, why only mbus_code? We have the same problem with mem2mem devices, where the format set on one queue affects the formats supported on another queue. Perhaps it should be defined to return formats valid with the current state of the driver? If we want to extend it to return formats for arbitrary states, perhaps we should use a mechanism similar to the TRY_FMT slot in subdevices, where we can set the configuration we want to test against and then ENUM_FMT would return the formats valid for that configuration?

Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

Good point.

We might want to keep the control'ification of the API in mind, which should simplify dealing with state inter-dependencies, because all the state would be represented in a uniform way.

I still don't know what Laurent wants to do with that.

Not sure why Laurent specifically. :)

On the other hand, that's a much more invasive change, so maybe better not to mix it with the things discussed here.

Quoted 29 lines
Quoted 20 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

Other improvements that could be made is to return more information about the format (similar to struct v4l2_format_info). In particular v4l2_pixel_encoding and mem/comp_planes would be useful for userspace to know.

Quoted 7 lines
Quoted 4 lines

An alternative would be to go away from mixing mem_planes and pixelformats and just having the "planarity" queryable and configurable separately. The existing duplicated FourCCs would have to remain for compatibility reasons, though.

Interesting idea, but I don't know if this would make the API more or less confusing.

Yeah, this definitely needs more thought. My experience with working with many people trying to use V4L2 in the userspace tells me that the existing model is confusing, though. DRM and most userspace libraries use FourCCs only to define the pixelformats themselves and planarity is a separate aspect.

Quoted 8 lines

The target model that I'd see happening would be to have the multiple-memory plane semantics used everywhere, so all color planes are described as separate entities, up to having different DMA-buf FDs. Then the single memory plane case would be just one of the specific cases, where all the DMA-buf FDs point to the same buffer and color plane offsets are laid out contiguously, which could be enforced by generic code when queuing the buffer if that's a hardware requirement.

Quoted 1 line

Do you think you can come up with a rough proposal before the ELCE session?

Sorry, I wasn't able to come up with anything formal, but I believe we discussed this well during the session and it's written down in the notes.

Best regards,
Tomasz

Quoted 24 lines

Regards,

Hans

Quoted 18 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

Finally, we can also add a new ioctl that combines ENUM_FMT/ENUM_FRAMESIZES/ENUM_FRAMEINTERVALS and returns an array of all valid formats/sizes/intervals, requiring just a single ioctl to obtain all this information.

Quoted 5 lines
Quoted 3 lines

While it definitely sounds like a useful thing to have, it would be an interesting problem to solve given the inter-dependencies between pads, queues and other state, as in the mbus example above.

This is definitely a separate issue for further in the future.

Agreed.

Best regards,
Tomasz

#8 lore raw

Re: [RFC PATCH v3 1/6] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more)

In reply to Boris Brezillon

Hi Boris,

Quoted 41 lines · Boris Brezillon

This is part of the multiplanar and singleplanar unification process. v4l2_ext_pix_format is supposed to work for both cases.

We also add the concept of modifiers already employed in DRM to expose HW-specific formats (like tiled or compressed formats) and allow exchanging this information with the DRM subsystem in a consistent way.

Note that V4L2_BUF_TYPE_VIDEO[_OUTPUT]_OVERLAY and V4L2_BUF_TYPE_VIDEO_{CAPTURE,OUTPUT}_MPLANE types are no longer accepted in v4l2_ext_format and will be rejected if you use the {G,S,TRY}EXT_FMT ioctls. V4L2_BUF_TYPE_VIDEO_{CAPTURE,OUTPUT}_MPLANE is dropped as part of the multiplanar/singleplanar unification. V4L2_BUF_TYPE_VIDEO[_OUTPUT]_OVERLAY seems to be used mostly on old drivers and supporting it would require some extra rework.

New hooks have been added to v4l2_ioctl_ops to support those new ioctls in drivers, but, in the meantime, the core takes care of converting {S,G,TRY}_EXT_FMT requests into {S,G,TRY}_FMT so that old drivers can still work if the userspace app/lib uses the new ioctls.
The conversion is also done the other around to allow userspace apps/libs using {S,G,TRY}_FMT to work with drivers implementing the _ext_ hooks.

---
Changes in v3:
- Rebased on top of media/master (post 5.4-rc1)

Changes in v2:
- Move the modifier in v4l2_ext_format (was formerly placed in v4l2_ext_plane)
- Fix a few bugs in the converters and add a strict parameter to allow conversion of uninitialized/mis-initialized objects

---
 drivers/media/v4l2-core/v4l2-dev.c   |  24 +-
 drivers/media/v4l2-core/v4l2-ioctl.c | 699 ++++++++++++++++++++++++---
 include/media/v4l2-ioctl.h           |  33 ++
 include/uapi/linux/videodev2.h       |  81 ++++
 4 files changed, 754 insertions(+), 83 deletions(-)

<snip>

Quoted 4 lines
drivers/media/v4l2-core/v4l2-ioctl.c +0 -0
index 51b912743f0f..78e14c1dc76f 100644

<snip>

Quoted 13 lines
drivers/media/v4l2-core/v4l2-ioctl.c +9 -0
@@ -1044,6 +1128,197 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
               sizeof(fmt->fmt.pix) - offset);
 }
 
+int v4l2_ext_format_to_format(const struct v4l2_ext_format *e,
+                             struct v4l2_format *f, bool mplane_cap,
+                             bool strict)
+{
+       const struct v4l2_plane_ext_pix_format *pe;
+       struct v4l2_plane_pix_format *p;
+       unsigned int i;
+
+       memset(f, 0, sizeof(*f));

drivers/media/v4l2-core/v4l2-ioctl.c static void v4l_sanitize_format(struct v4l2_format *fmt) · line 1149

Quoted 10 lines
drivers/media/v4l2-core/v4l2-ioctl.c +10 -0
+
+       switch (e->type) {
+       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+               /*
+                * Make sure no modifier is required before doing the
+                * conversion.
+                */
+               if (e->fmt.pix.modifier && strict)
+                       return -EINVAL;

In my driver, I set the modifier of regular pixel formats to DRM_FORMAT_MOD_LINEAR, which conflicted with this check.

IMHO this check should be dropped entirely in case userspace is already handling the modifiers itself using the previous V4L format API.

Quoted 5 lines
drivers/media/v4l2-core/v4l2-ioctl.c +5 -0
+
+               if ((e->fmt.pix.num_planes > VIDEO_MAX_PLANES ||
+                    !e->fmt.pix.num_planes) && strict)
+                       return -EINVAL;
+

<snip>

drivers/media/v4l2-core/v4l2-ioctl.c static int v4l_enum_fmt(const struct v4l2_ioctl_ops *ops, · line 1749

Quoted 10 lines
drivers/media/v4l2-core/v4l2-ioctl.c +6 -0
@@ -1466,6 +1741,38 @@ static int v4l_enum_fmt(const struct v4l2_ioctl_ops *ops,
        return ret;
 }
 
+static int v4l_g_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
+                            struct file *file, void *fh,
+                            struct v4l2_format *f)
+{
+       struct v4l2_ext_format ef = {
+               .type = f->type,

This can set ef.type to an invalid value (V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE or V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) which will have adverse effects later on.

Quoted 19 lines
drivers/media/v4l2-core/v4l2-ioctl.c +19 -0
+       };
+       int ret;
+
+       switch (f->type) {
+       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+       case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
+               ret = ops->vidioc_g_ext_fmt_vid_cap(file, fh, &ef.fmt.pix);
+               break;
+
+       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+       case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
+               ret = ops->vidioc_g_ext_fmt_vid_out(file, fh, &ef.fmt.pix);
+               break;
+
+       default:
+               return -EINVAL;
+       }
+
+       if (ret)

drivers/media/v4l2-core/v4l2-ioctl.c static int v4l_enum_fmt(const struct v4l2_ioctl_ops *ops, · line 1778

Quoted 10 lines
drivers/media/v4l2-core/v4l2-ioctl.c +7 -0
+               return ret;
+
+       return v4l2_ext_format_to_format(&ef, f,
+                                        V4L2_TYPE_IS_MULTIPLANAR(f->type),
+                                        true);
+}
+
 static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
                                struct file *file, void *fh, void *arg)
 {

<snip>

I picked up on this patch since I wanted to add DRM modifiers to the format API, so that we could do a V4L -> Userspace -> DRM pipeline that included modifiers from start to end. I was glad to find yours, thanks!
Overall the driver transition was very smooth and apart from the 2 issues I encountered, userspace (ffmpeg using v4l2 decoding in this case) kept working nicely using the previous format API.

Regarding ENUM_FMT, I feel like it should expose DRM modifiers as well. The driver would fill that field with all possible combinations (xor'd) of modifiers for a specific pixfmt.

Regards,
Maxime

#9 lore raw

Re: [RFC PATCH v3 1/6] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more)

Quoted 68 lines · On Mon, 24 Feb 2020 15:17:31 +0100 Maxime Jourdan <[email protected]> wrote:

Hi Boris,

Quoted 41 lines · Boris Brezillon

This is part of the multiplanar and singleplanar unification process. v4l2_ext_pix_format is supposed to work for both cases.

We also add the concept of modifiers already employed in DRM to expose HW-specific formats (like tiled or compressed formats) and allow exchanging this information with the DRM subsystem in a consistent way.

Note that V4L2_BUF_TYPE_VIDEO[_OUTPUT]_OVERLAY and V4L2_BUF_TYPE_VIDEO_{CAPTURE,OUTPUT}_MPLANE types are no longer accepted in v4l2_ext_format and will be rejected if you use the {G,S,TRY}EXT_FMT ioctls. V4L2_BUF_TYPE_VIDEO_{CAPTURE,OUTPUT}_MPLANE is dropped as part of the multiplanar/singleplanar unification. V4L2_BUF_TYPE_VIDEO[_OUTPUT]_OVERLAY seems to be used mostly on old drivers and supporting it would require some extra rework.

New hooks have been added to v4l2_ioctl_ops to support those new ioctls in drivers, but, in the meantime, the core takes care of converting {S,G,TRY}_EXT_FMT requests into {S,G,TRY}_FMT so that old drivers can still work if the userspace app/lib uses the new ioctls.
The conversion is also done the other around to allow userspace apps/libs using {S,G,TRY}_FMT to work with drivers implementing the _ext_ hooks.

---
Changes in v3:
- Rebased on top of media/master (post 5.4-rc1)

Changes in v2:
- Move the modifier in v4l2_ext_format (was formerly placed in v4l2_ext_plane)
- Fix a few bugs in the converters and add a strict parameter to allow conversion of uninitialized/mis-initialized objects

---
 drivers/media/v4l2-core/v4l2-dev.c   |  24 +-
 drivers/media/v4l2-core/v4l2-ioctl.c | 699 ++++++++++++++++++++++++---
 include/media/v4l2-ioctl.h           |  33 ++
 include/uapi/linux/videodev2.h       |  81 ++++
 4 files changed, 754 insertions(+), 83 deletions(-)

<snip>

Quoted 4 lines
drivers/media/v4l2-core/v4l2-ioctl.c +0 -0
index 51b912743f0f..78e14c1dc76f 100644

<snip>

Quoted 18 lines
drivers/media/v4l2-core/v4l2-ioctl.c +14 -0
@@ -1044,6 +1128,197 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
               sizeof(fmt->fmt.pix) - offset);
 }
 
+int v4l2_ext_format_to_format(const struct v4l2_ext_format *e,
+                             struct v4l2_format *f, bool mplane_cap,
+                             bool strict)
+{
+       const struct v4l2_plane_ext_pix_format *pe;
+       struct v4l2_plane_pix_format *p;
+       unsigned int i;
+
+       memset(f, 0, sizeof(*f));
+
+       switch (e->type) {
+       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+               /*
Quoted 8 lines

drivers/media/v4l2-core/v4l2-ioctl.c static void v4l_sanitize_format(struct v4l2_format *fmt) · line 1149

Quoted 5 lines
drivers/media/v4l2-core/v4l2-ioctl.c +5 -0
+                * Make sure no modifier is required before doing the
+                * conversion.
+                */
+               if (e->fmt.pix.modifier && strict)
+                       return -EINVAL;  

In my driver, I set the modifier of regular pixel formats to DRM_FORMAT_MOD_LINEAR, which conflicted with this check.

I guess we could check for != MOD_LINEAR && != MOD_INVALID.

Quoted 4 lines

IMHO this check should be dropped entirely in case userspace is already handling the modifiers itself using the previous V4L format API.

I'm not sure. Shouldn't we catch cases where the ext -> !ext format conversion implies information loss?

Quoted 11 lines
Quoted 6 lines
drivers/media/v4l2-core/v4l2-ioctl.c +5 -0
+
+               if ((e->fmt.pix.num_planes > VIDEO_MAX_PLANES ||
+                    !e->fmt.pix.num_planes) && strict)
+                       return -EINVAL;
+  

<snip>

Quoted 4 lines
drivers/media/v4l2-core/v4l2-ioctl.c +0 -0
@@ -1466,6 +1741,38 @@ static int v4l_enum_fmt(const struct v4l2_ioctl_ops *ops,
        return ret;
 }
 
Quoted 11 lines

drivers/media/v4l2-core/v4l2-ioctl.c static int v4l_enum_fmt(const struct v4l2_ioctl_ops *ops, · line 1749

Quoted 6 lines
drivers/media/v4l2-core/v4l2-ioctl.c +6 -0
+static int v4l_g_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
+                            struct file *file, void *fh,
+                            struct v4l2_format *f)
+{
+       struct v4l2_ext_format ef = {
+               .type = f->type,  

This can set ef.type to an invalid value (V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE or V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) which will have adverse effects later on.

Oops, indeed. MPLANE formats should be reduced to their !MPLANE equivalent.

Quoted 40 lines
Quoted 20 lines
drivers/media/v4l2-core/v4l2-ioctl.c +19 -0
+       };
+       int ret;
+
+       switch (f->type) {
+       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+       case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
+               ret = ops->vidioc_g_ext_fmt_vid_cap(file, fh, &ef.fmt.pix);
+               break;
+
+       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+       case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
+               ret = ops->vidioc_g_ext_fmt_vid_out(file, fh, &ef.fmt.pix);
+               break;
+
+       default:
+               return -EINVAL;
+       }
+
+       if (ret)

drivers/media/v4l2-core/v4l2-ioctl.c static int v4l_enum_fmt(const struct v4l2_ioctl_ops *ops, · line 1778

Quoted 10 lines
drivers/media/v4l2-core/v4l2-ioctl.c +7 -0
+               return ret;
+
+       return v4l2_ext_format_to_format(&ef, f,
+                                        V4L2_TYPE_IS_MULTIPLANAR(f->type),
+                                        true);
+}
+
 static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
                                struct file *file, void *fh, void *arg)
 {  

<snip>

I picked up on this patch since I wanted to add DRM modifiers to the format API, so that we could do a V4L -> Userspace -> DRM pipeline that included modifiers from start to end. I was glad to find yours, thanks!
Overall the driver transition was very smooth and apart from the 2 issues I encountered, userspace (ffmpeg using v4l2 decoding in this case) kept working nicely using the previous format API.

Quoted 3 lines

Regarding ENUM_FMT, I feel like it should expose DRM modifiers as well. The driver would fill that field with all possible combinations (xor'd) of modifiers for a specific pixfmt.

You might want to have a look at [1] (a few adjustments have been suggested during the last media summit).

[1]https://lore.kernel.org/linux-media/CAAFQd5AN1nCWwtGVRp0BasQHZvXwesMx4g2FU=KL4ZyZzPq81g@mail.gmail.com/T/

#10 lore raw

RFC PATCH v3 2/6 media: v4l2: Add extended buffer operations

In reply to Boris Brezillon

From: Hans Verkuil <[email protected]>

Those extended buffer ops have several purpose:
1/ Fix y2038 issues by converting the timestamp into an u64 counting
the number of ns elapsed since 1970
2/ Unify single/multiplanar handling
3/ Add a new start offset field to each v4l2 plane buffer info struct
to support the case where a single buffer object is storing all planes data, each one being placed at a different offset

New hooks are created in v4l2_ioctl_ops so that drivers can start using these new objects.

The core takes care of converting new ioctls requests to old ones if the driver does not support the new hooks, and vice versa.

Note that the timecode field is gone, since there doesn't seem to be in-kernel users, but can be added back in the reserved area if needed.

---
Changes in v3:
- Rebased on top of media/master (post 5.4-rc1)

Changes in v2:
- Add reserved space to v4l2_ext_buffer so that new fields can be added later on

---
 drivers/media/v4l2-core/v4l2-dev.c   |  30 +-
 drivers/media/v4l2-core/v4l2-ioctl.c | 428 +++++++++++++++++++++++++--
 include/media/v4l2-ioctl.h           |  30 ++
 include/uapi/linux/videodev2.h       | 130 ++++++++
 4 files changed, 591 insertions(+), 27 deletions(-)
drivers/media/v4l2-core/v4l2-dev.c +24 -6
index 9aad715537b3..35c8caccd025 100644
@@ -696,12 +696,30 @@ static void determine_valid_ioctls(struct video_device *vdev)
 	if (is_vid || is_vbi || is_sdr || is_tch) {
 		/* ioctls valid for video, metadata, vbi or sdr */
 		SET_VALID_IOCTL(ops, VIDIOC_REQBUFS, vidioc_reqbufs);
-		SET_VALID_IOCTL(ops, VIDIOC_QUERYBUF, vidioc_querybuf);
-		SET_VALID_IOCTL(ops, VIDIOC_QBUF, vidioc_qbuf);
-		SET_VALID_IOCTL(ops, VIDIOC_EXPBUF, vidioc_expbuf);
-		SET_VALID_IOCTL(ops, VIDIOC_DQBUF, vidioc_dqbuf);
-		SET_VALID_IOCTL(ops, VIDIOC_CREATE_BUFS, vidioc_create_bufs);
-		SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, vidioc_prepare_buf);
+		if (ops->vidioc_querybuf || ops->vidioc_ext_querybuf) {
+			set_bit(_IOC_NR(VIDIOC_QUERYBUF), valid_ioctls);
+			set_bit(_IOC_NR(VIDIOC_EXT_QUERYBUF), valid_ioctls);
+		}
+		if (ops->vidioc_qbuf || ops->vidioc_ext_qbuf) {
+			set_bit(_IOC_NR(VIDIOC_QBUF), valid_ioctls);
+			set_bit(_IOC_NR(VIDIOC_EXT_QBUF), valid_ioctls);
+		}
+		if (ops->vidioc_expbuf || ops->vidioc_ext_expbuf) {
+			set_bit(_IOC_NR(VIDIOC_EXPBUF), valid_ioctls);
+			set_bit(_IOC_NR(VIDIOC_EXT_EXPBUF), valid_ioctls);
+		}
+		if (ops->vidioc_dqbuf || ops->vidioc_ext_dqbuf) {
+			set_bit(_IOC_NR(VIDIOC_DQBUF), valid_ioctls);
+			set_bit(_IOC_NR(VIDIOC_EXT_DQBUF), valid_ioctls);
+		}
+		if (ops->vidioc_create_bufs || ops->vidioc_ext_create_bufs) {
+			set_bit(_IOC_NR(VIDIOC_CREATE_BUFS), valid_ioctls);
+			set_bit(_IOC_NR(VIDIOC_EXT_CREATE_BUFS), valid_ioctls);
+		}
+		if (ops->vidioc_prepare_buf || ops->vidioc_ext_prepare_buf) {
+			set_bit(_IOC_NR(VIDIOC_PREPARE_BUF), valid_ioctls);
+			set_bit(_IOC_NR(VIDIOC_EXT_PREPARE_BUF), valid_ioctls);
+		}
 		SET_VALID_IOCTL(ops, VIDIOC_STREAMON, vidioc_streamon);
 		SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, vidioc_streamoff);
 	}
drivers/media/v4l2-core/v4l2-ioctl.c +407 -21

514 lines not on this page · raw message

include/media/v4l2-ioctl.h +30 -0
index 39ac07fbc7b7..f7e375d38602 100644
@@ -168,16 +168,28 @@ struct v4l2_fh;
  *	:ref:`VIDIOC_REQBUFS <vidioc_reqbufs>` ioctl
  * @vidioc_querybuf: pointer to the function that implements
  *	:ref:`VIDIOC_QUERYBUF <vidioc_querybuf>` ioctl
+ * @vidioc_ext_querybuf: pointer to the function that implements
+ *	:ref:`VIDIOC_EXT_QUERYBUF <vidioc_ext_querybuf>` ioctl
  * @vidioc_qbuf: pointer to the function that implements
  *	:ref:`VIDIOC_QBUF <vidioc_qbuf>` ioctl
+ * @vidioc_ext_qbuf: pointer to the function that implements
+ *	:ref:`VIDIOC_EXT_QBUF <vidioc_ext_qbuf>` ioctl
  * @vidioc_expbuf: pointer to the function that implements
  *	:ref:`VIDIOC_EXPBUF <vidioc_expbuf>` ioctl
+ * @vidioc_ext_expbuf: pointer to the function that implements
+ *	:ref:`VIDIOC_EXT_EXPBUF <vidioc_ext_expbuf>` ioctl
  * @vidioc_dqbuf: pointer to the function that implements
  *	:ref:`VIDIOC_DQBUF <vidioc_qbuf>` ioctl
+ * @vidioc_ext_dqbuf: pointer to the function that implements
+ *	:ref:`VIDIOC_EXT_DQBUF <vidioc_ext_qbuf>` ioctl
  * @vidioc_create_bufs: pointer to the function that implements
  *	:ref:`VIDIOC_CREATE_BUFS <vidioc_create_bufs>` ioctl
+ * @vidioc_ext_create_bufs: pointer to the function that implements
+ *	:ref:`VIDIOC_EXT_CREATE_BUFS <vidioc_ext_create_bufs>` ioctl
  * @vidioc_prepare_buf: pointer to the function that implements
  *	:ref:`VIDIOC_PREPARE_BUF <vidioc_prepare_buf>` ioctl
+ * @vidioc_ext_prepare_buf: pointer to the function that implements
+ *	:ref:`VIDIOC_EXT_PREPARE_BUF <vidioc_ext_prepare_buf>` ioctl
  * @vidioc_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_OVERLAY <vidioc_overlay>` ioctl
  * @vidioc_g_fbuf: pointer to the function that implements
@@ -438,17 +450,29 @@ struct v4l2_ioctl_ops {
 			      struct v4l2_requestbuffers *b);
 	int (*vidioc_querybuf)(struct file *file, void *fh,
 			       struct v4l2_buffer *b);
+	int (*vidioc_ext_querybuf)(struct file *file, void *fh,
+				   struct v4l2_ext_buffer *b);
 	int (*vidioc_qbuf)(struct file *file, void *fh,
 			   struct v4l2_buffer *b);
+	int (*vidioc_ext_qbuf)(struct file *file, void *fh,
+			       struct v4l2_ext_buffer *b);
 	int (*vidioc_expbuf)(struct file *file, void *fh,
 			     struct v4l2_exportbuffer *e);
+	int (*vidioc_ext_expbuf)(struct file *file, void *fh,
+				 struct v4l2_ext_exportbuffer *e);
 	int (*vidioc_dqbuf)(struct file *file, void *fh,
 			    struct v4l2_buffer *b);
+	int (*vidioc_ext_dqbuf)(struct file *file, void *fh,
+				struct v4l2_ext_buffer *b);
 
 	int (*vidioc_create_bufs)(struct file *file, void *fh,
 				  struct v4l2_create_buffers *b);
+	int (*vidioc_ext_create_bufs)(struct file *file, void *fh,
+				      struct v4l2_ext_create_buffers *b);
 	int (*vidioc_prepare_buf)(struct file *file, void *fh,
 				  struct v4l2_buffer *b);
+	int (*vidioc_ext_prepare_buf)(struct file *file, void *fh,
+				      struct v4l2_ext_buffer *b);
 
 	int (*vidioc_overlay)(struct file *file, void *fh, unsigned int i);
 	int (*vidioc_g_fbuf)(struct file *file, void *fh,
@@ -757,4 +781,10 @@ int v4l2_ext_format_to_format(const struct v4l2_ext_format *e,
 			      struct v4l2_format *f,
 			      bool mplane_cap, bool strict);
 
+int v4l2_ext_buffer_to_buffer(const struct v4l2_ext_buffer *e,
+			      struct v4l2_buffer *b,
+			      bool mplane_cap);
+int v4l2_buffer_to_ext_buffer(const struct v4l2_buffer *b,
+			      struct v4l2_ext_buffer *e);
+
 #endif /* _V4L2_IOCTL_H */
include/uapi/linux/videodev2.h +130 -0

160 lines not on this page · raw message

Signature · 1 line
2.21.0
#11 lore raw

Re: [RFC PATCH 3/3] media: v4l2: Add extended buffer operations

Thanks for putting this patch set together. I'm currently trying to get modifiers working for UBWC. I've applied these patches and with a little tweaking can get video decode working without modifiers. I haven't tried the new ioctls yet.

Quoted 144 lines

From: Hans Verkuil <[email protected]>

Those extended buffer ops have several purpose:
1/ Fix y2038 issues by converting the timestamp into an u64 counting
the number of ns elapsed since 1970
2/ Unify single/multiplanar handling
3/ Add a new start offset field to each v4l2 plane buffer info struct
to support the case where a single buffer object is storing all planes data, each one being placed at a different offset

New hooks are created in v4l2_ioctl_ops so that drivers can start using these new objects.

The core takes care of converting new ioctls requests to old ones if the driver does not support the new hooks, and vice versa.

Note that the timecode field is gone, since there doesn't seem to be in-kernel users, but can be added back in the reserved area if needed.

---
Changes in v3:
- Rebased on top of media/master (post 5.4-rc1)

Changes in v2:
- Add reserved space to v4l2_ext_buffer so that new fields can be added later on

---
 drivers/media/v4l2-core/v4l2-dev.c   |  30 +-
 drivers/media/v4l2-core/v4l2-ioctl.c | 428 +++++++++++++++++++++++++--
 include/media/v4l2-ioctl.h           |  30 ++
 include/uapi/linux/videodev2.h       | 130 ++++++++
 4 files changed, 591 insertions(+), 27 deletions(-)
drivers/media/v4l2-core/v4l2-dev.c +24 -6
index 9aad715537b3..35c8caccd025 100644
@@ -696,12 +696,30 @@ static void determine_valid_ioctls(struct video_device *vdev)
 	if (is_vid || is_vbi || is_sdr || is_tch) {
 		/* ioctls valid for video, metadata, vbi or sdr */
 		SET_VALID_IOCTL(ops, VIDIOC_REQBUFS, vidioc_reqbufs);
-		SET_VALID_IOCTL(ops, VIDIOC_QUERYBUF, vidioc_querybuf);
-		SET_VALID_IOCTL(ops, VIDIOC_QBUF, vidioc_qbuf);
-		SET_VALID_IOCTL(ops, VIDIOC_EXPBUF, vidioc_expbuf);
-		SET_VALID_IOCTL(ops, VIDIOC_DQBUF, vidioc_dqbuf);
-		SET_VALID_IOCTL(ops, VIDIOC_CREATE_BUFS, vidioc_create_bufs);
-		SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, vidioc_prepare_buf);
+		if (ops->vidioc_querybuf || ops->vidioc_ext_querybuf) {
+			set_bit(_IOC_NR(VIDIOC_QUERYBUF), valid_ioctls);
+			set_bit(_IOC_NR(VIDIOC_EXT_QUERYBUF), valid_ioctls);
+		}
+		if (ops->vidioc_qbuf || ops->vidioc_ext_qbuf) {
+			set_bit(_IOC_NR(VIDIOC_QBUF), valid_ioctls);
+			set_bit(_IOC_NR(VIDIOC_EXT_QBUF), valid_ioctls);
+		}
+		if (ops->vidioc_expbuf || ops->vidioc_ext_expbuf) {
+			set_bit(_IOC_NR(VIDIOC_EXPBUF), valid_ioctls);
+			set_bit(_IOC_NR(VIDIOC_EXT_EXPBUF), valid_ioctls);
+		}
+		if (ops->vidioc_dqbuf || ops->vidioc_ext_dqbuf) {
+			set_bit(_IOC_NR(VIDIOC_DQBUF), valid_ioctls);
+			set_bit(_IOC_NR(VIDIOC_EXT_DQBUF), valid_ioctls);
+		}
+		if (ops->vidioc_create_bufs || ops->vidioc_ext_create_bufs) {
+			set_bit(_IOC_NR(VIDIOC_CREATE_BUFS), valid_ioctls);
+			set_bit(_IOC_NR(VIDIOC_EXT_CREATE_BUFS), valid_ioctls);
+		}
+		if (ops->vidioc_prepare_buf || ops->vidioc_ext_prepare_buf) {
+			set_bit(_IOC_NR(VIDIOC_PREPARE_BUF), valid_ioctls);
+			set_bit(_IOC_NR(VIDIOC_EXT_PREPARE_BUF), valid_ioctls);
+		}
 		SET_VALID_IOCTL(ops, VIDIOC_STREAMON, vidioc_streamon);
 		SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, vidioc_streamoff);
 	}
drivers/media/v4l2-core/v4l2-ioctl.c +40 -0
index 78e14c1dc76f..356218e44ccb 100644
@@ -579,6 +579,25 @@ static void v4l_print_buffer(const void *arg, bool write_only)
 			tc->type, tc->flags, tc->frames, *(__u32 *)tc->userbits);
 }
 
+static void v4l_print_ext_buffer(const void *arg, bool write_only)
+{
+	const struct v4l2_ext_buffer *p = arg;
+	const struct v4l2_ext_plane *plane;
+	int i;
+
+	pr_cont("%lld index=%d, type=%s, flags=0x%08x, field=%s, sequence=%d, memory=%s\n",
+		p->timestamp, p->index, prt_names(p->type, v4l2_type_names),
+		p->flags, prt_names(p->field, v4l2_field_names),
+		p->sequence, prt_names(p->memory, v4l2_memory_names));
+
+	for (i = 0; i < p->num_planes; ++i) {
+		plane = &p->planes[i];
+		pr_debug("plane %d: bytesused=%d, data_offset=0x%08x, offset/userptr=0x%llx, length=%d\n",
+			 i, plane->bytesused, plane->data_offset,
+			 plane->m.userptr, plane->length);
+	}
+}
+
 static void v4l_print_exportbuffer(const void *arg, bool write_only)
 {
 	const struct v4l2_exportbuffer *p = arg;
@@ -588,6 +607,18 @@ static void v4l_print_exportbuffer(const void *arg, bool write_only)
 		p->index, p->plane, p->flags);
 }
 
+static void v4l_print_ext_exportbuffer(const void *arg, bool write_only)
+{
+	const struct v4l2_ext_exportbuffer *p = arg;
+	unsigned int i;
+
+	pr_cont("type=%s, index=%u, first_plane=%u num_planes=%u, flags=%08x\n",
+		prt_names(p->type, v4l2_type_names), p->index, p->first_plane,
+		p->num_planes, p->flags);
+	for (i = p->first_plane; i < p->first_plane + p->num_planes; ++i)
+		pr_debug("plane %u: fd=%d\n", i, p->fds[i]);
+}
+
 static void v4l_print_create_buffers(const void *arg, bool write_only)
 {
 	const struct v4l2_create_buffers *p = arg;
@@ -598,6 +629,15 @@ static void v4l_print_create_buffers(const void *arg, bool write_only)
 	v4l_print_format(&p->format, write_only);
 }
 
+static void v4l_print_ext_create_buffers(const void *arg, bool write_only)
+{
+	const struct v4l2_ext_create_buffers *p = arg;
+
+	pr_cont("index=%d, count=%d, memory=%s, ", p->index, p->count,
+		prt_names(p->memory, v4l2_memory_names));
+	v4l_print_ext_format(&p->format, write_only);
+}
+
 static void v4l_print_streamparm(const void *arg, bool write_only)
 {
 	const struct v4l2_streamparm *p = arg;
@@ -1319,6 +1359,123 @@ int v4l2_format_to_ext_format(const struct v4l2_format *f,
 }
 EXPORT_SYMBOL_GPL(v4l2_format_to_ext_format);

drivers/media/v4l2-core/v4l2-ioctl.c int v4l2_format_to_ext_format(const struct v4l2_format *f, · line 1370

Quoted 10 lines
drivers/media/v4l2-core/v4l2-ioctl.c +9 -0
 
+int v4l2_ext_buffer_to_buffer(const struct v4l2_ext_buffer *e,
+			      struct v4l2_buffer *b, bool mplane_cap)
+{
+	u64 nsecs;
+
+	if (!mplane_cap && e->num_planes > 1)
+		return -EINVAL;
+
+	memset(b, 0, sizeof(*b));

b->planes needs to be preserved.

I was not able to find a tree that these patches applied to cleanly. In v5.4 the struct v4l2_buffer member planes is a raw pointer, so this memset will clear it out.

Quoted 698 lines
drivers/media/v4l2-core/v4l2-ioctl.c +358 -21
+
+	b->index = e->index;
+	b->flags = e->flags;
+	b->field = e->field;
+	b->sequence = e->sequence;
+	b->memory = e->memory;
+	b->request_fd = e->request_fd;
+	b->timestamp.tv_sec = div64_u64_rem(e->timestamp, NSEC_PER_SEC, &nsecs);
+	b->timestamp.tv_usec = (u32)nsecs / NSEC_PER_USEC;
+	if (mplane_cap) {
+		unsigned int i;
+
+		if (e->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
+			b->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
+		else
+			b->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
+
+		b->length = e->num_planes;
+		for (i = 0; i < e->num_planes; i++) {
+			if (b->memory == V4L2_MEMORY_DMABUF) {
+				if (e->planes[i].m.dmabuf.offset)
+					return -EINVAL;
+
+				b->m.planes[i].m.fd = e->planes[i].m.dmabuf.fd;
+			} else {
+				b->m.planes[i].m.userptr = e->planes[i].m.userptr;
+			}
+			b->m.planes[i].length = e->planes[i].length;
+			b->m.planes[i].bytesused = e->planes[i].bytesused;
+			b->m.planes[i].data_offset = e->planes[i].data_offset;
+			memset(b->m.planes[i].reserved, 0,
+			       sizeof(b->m.planes[i].reserved));
+		}
+	} else {
+		b->type = e->type;
+		b->bytesused = e->planes[0].bytesused;
+		b->length = e->planes[0].length;
+		if (b->memory == V4L2_MEMORY_DMABUF) {
+			if (e->planes[0].m.dmabuf.offset)
+				return -EINVAL;
+
+			b->m.fd = e->planes[0].m.dmabuf.fd;
+		} else {
+			b->m.userptr = e->planes[0].m.userptr;
+		}
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_ext_buffer_to_buffer);
+
+int v4l2_buffer_to_ext_buffer(const struct v4l2_buffer *b,
+			      struct v4l2_ext_buffer *e)
+{
+	memset(e, 0, sizeof(*e));
+
+	e->index = b->index;
+	e->flags = b->flags;
+	e->field = b->field;
+	e->sequence = b->sequence;
+	e->memory = b->memory;
+	e->request_fd = b->request_fd;
+	e->timestamp = b->timestamp.tv_sec * NSEC_PER_SEC +
+		b->timestamp.tv_usec * NSEC_PER_USEC;
+	if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
+		unsigned int i;
+
+		if (!b->m.planes)
+			return -EINVAL;
+
+		if (b->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
+			e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+		else
+			e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
+
+		e->num_planes = b->length;
+		for (i = 0; i < e->num_planes; i++) {
+			if (b->memory == V4L2_MEMORY_DMABUF) {
+				e->planes[i].m.dmabuf.fd = b->m.planes[i].m.fd;
+				e->planes[i].m.dmabuf.offset = 0;
+			} else {
+				e->planes[i].m.userptr = b->m.planes[i].m.userptr;
+			}
+			e->planes[i].length = b->m.planes[i].length;
+			e->planes[i].bytesused = b->m.planes[i].bytesused;
+			e->planes[i].data_offset = b->m.planes[i].data_offset;
+			memset(e->planes[i].reserved, 0,
+			       sizeof(e->planes[i].reserved));
+		}
+	} else {
+		e->type = b->type;
+		e->num_planes = 1;
+		e->planes[0].bytesused = b->bytesused;
+		e->planes[0].length = b->length;
+		if (b->memory == V4L2_MEMORY_DMABUF) {
+			e->planes[0].m.dmabuf.fd = b->m.fd;
+			e->planes[0].m.dmabuf.offset = 0;
+		} else {
+			e->planes[0].m.userptr = b->m.userptr;
+		}
+		e->planes[0].m.userptr = b->m.userptr;
+		e->planes[0].data_offset = 0;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_buffer_to_ext_buffer);
+
 static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
 				struct file *file, void *fh, void *arg)
 {
@@ -2506,31 +2663,109 @@ static int v4l_reqbufs(const struct v4l2_ioctl_ops *ops,
 	return ops->vidioc_reqbufs(file, fh, p);
 }
 
-static int v4l_querybuf(const struct v4l2_ioctl_ops *ops,
-				struct file *file, void *fh, void *arg)
+static int v4l_do_buf_op(int (*op)(struct file *, void *,
+				   struct v4l2_buffer *),
+			 int (*ext_op)(struct file *, void *,
+				       struct v4l2_ext_buffer *),
+			 struct file *file, void *fh, struct v4l2_buffer *b)
 {
-	struct v4l2_buffer *p = arg;
-	int ret = check_fmt(file, p->type);
+	struct v4l2_ext_buffer eb;
+	int ret;
 
-	return ret ? ret : ops->vidioc_querybuf(file, fh, p);
+	ret = check_fmt(file, b->type);
+	if (ret)
+		return ret;
+
+	if (op)
+		return op(file, fh, b);
+
+	ret = v4l2_buffer_to_ext_buffer(b, &eb);
+	if (ret)
+		return ret;
+
+	ret = ext_op(file, fh, &eb);
+	if (ret)
+		return ret;
+
+	v4l2_ext_buffer_to_buffer(&eb, b, V4L2_TYPE_IS_MULTIPLANAR(b->type));
+	return 0;
+}
+
+static int v4l_do_ext_buf_op(int (*op)(struct file *, void *,
+				       struct v4l2_buffer *),
+			     int (*ext_op)(struct file *, void *,
+					   struct v4l2_ext_buffer *),
+			     struct file *file, void *fh,
+			     struct v4l2_ext_buffer *eb)
+{
+	struct video_device *vdev = video_devdata(file);
+	struct v4l2_buffer b;
+	bool mplane_cap;
+	int ret;
+
+	ret = check_fmt(file, eb->type);
+	if (ret)
+		return ret;
+
+	if (ext_op)
+		return ext_op(file, fh, eb);
+
+	mplane_cap = !!(vdev->device_caps &
+			(V4L2_CAP_VIDEO_CAPTURE_MPLANE |
+			 V4L2_CAP_VIDEO_OUTPUT_MPLANE |
+			 V4L2_CAP_VIDEO_M2M_MPLANE));
+	ret = v4l2_ext_buffer_to_buffer(eb, &b, mplane_cap);
+	if (ret)
+		return ret;
+
+	ret = op(file, fh, &b);
+	if (ret)
+		return ret;
+
+	v4l2_buffer_to_ext_buffer(&b, eb);
+	return 0;
+}
+
+static int v4l_querybuf(const struct v4l2_ioctl_ops *ops,
+			struct file *file, void *fh, void *arg)
+{
+	return v4l_do_buf_op(ops->vidioc_querybuf, ops->vidioc_ext_querybuf,
+			     file, fh, arg);
+}
+
+static int v4l_ext_querybuf(const struct v4l2_ioctl_ops *ops,
+			    struct file *file, void *fh, void *arg)
+{
+	return v4l_do_ext_buf_op(ops->vidioc_querybuf,
+				 ops->vidioc_ext_querybuf, file, fh, arg);
 }
 
 static int v4l_qbuf(const struct v4l2_ioctl_ops *ops,
-				struct file *file, void *fh, void *arg)
+		    struct file *file, void *fh, void *arg)
 {
-	struct v4l2_buffer *p = arg;
-	int ret = check_fmt(file, p->type);
+	return v4l_do_buf_op(ops->vidioc_qbuf, ops->vidioc_ext_qbuf,
+			     file, fh, arg);
+}
 
-	return ret ? ret : ops->vidioc_qbuf(file, fh, p);
+static int v4l_ext_qbuf(const struct v4l2_ioctl_ops *ops,
+			struct file *file, void *fh, void *arg)
+{
+	return v4l_do_ext_buf_op(ops->vidioc_qbuf, ops->vidioc_ext_qbuf,
+				 file, fh, arg);
 }
 
 static int v4l_dqbuf(const struct v4l2_ioctl_ops *ops,
-				struct file *file, void *fh, void *arg)
+		     struct file *file, void *fh, void *arg)
 {
-	struct v4l2_buffer *p = arg;
-	int ret = check_fmt(file, p->type);
+	return v4l_do_buf_op(ops->vidioc_dqbuf, ops->vidioc_ext_dqbuf,
+			     file, fh, arg);
+}
 
-	return ret ? ret : ops->vidioc_dqbuf(file, fh, p);
+static int v4l_ext_dqbuf(const struct v4l2_ioctl_ops *ops,
+			 struct file *file, void *fh, void *arg)
+{
+	return v4l_do_ext_buf_op(ops->vidioc_dqbuf, ops->vidioc_ext_dqbuf,
+				 file, fh, arg);
 }
 
 static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
@@ -2546,7 +2781,27 @@ static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
 
 	v4l_sanitize_format(&create->format);
 
-	ret = ops->vidioc_create_bufs(file, fh, create);
+	if (ops->vidioc_create_bufs) {
+		ret = ops->vidioc_create_bufs(file, fh, create);
+	} else {
+		struct v4l2_ext_create_buffers ecreate = {
+			.count = create->count,
+			.memory = create->memory,
+		};
+
+		ret = v4l2_format_to_ext_format(&create->format,
+						&ecreate.format, true);
+		if (ret)
+			return ret;
+
+		ret = ops->vidioc_ext_create_bufs(file, fh, &ecreate);
+		if (ret)
+			return ret;
+
+		create->index = ecreate.index;
+		create->count = ecreate.count;
+		create->capabilities = ecreate.capabilities;
+	}
 
 	if (create->format.type == V4L2_BUF_TYPE_VIDEO_CAPTURE ||
 	    create->format.type == V4L2_BUF_TYPE_VIDEO_OUTPUT)
@@ -2555,13 +2810,59 @@ static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
 	return ret;
 }
 
-static int v4l_prepare_buf(const struct v4l2_ioctl_ops *ops,
-				struct file *file, void *fh, void *arg)
+static int v4l_ext_create_bufs(const struct v4l2_ioctl_ops *ops,
+			       struct file *file, void *fh, void *arg)
 {
-	struct v4l2_buffer *b = arg;
-	int ret = check_fmt(file, b->type);
+	struct v4l2_ext_create_buffers *ecreate = arg;
+	struct video_device *vdev = video_devdata(file);
+	struct v4l2_create_buffers create = {
+		.count = ecreate->count,
+		.memory = ecreate->memory,
+	};
+	bool mplane_cap;
+	int ret;
 
-	return ret ? ret : ops->vidioc_prepare_buf(file, fh, b);
+	ret = check_fmt(file, ecreate->format.type);
+	if (ret)
+		return ret;
+
+	if (ops->vidioc_ext_create_bufs)
+		return ops->vidioc_ext_create_bufs(file, fh, ecreate);
+
+	mplane_cap = !!(vdev->device_caps &
+			(V4L2_CAP_VIDEO_CAPTURE_MPLANE |
+			 V4L2_CAP_VIDEO_OUTPUT_MPLANE |
+			 V4L2_CAP_VIDEO_M2M_MPLANE));
+	ret = v4l2_ext_format_to_format(&ecreate->format,
+					&create.format, mplane_cap, true);
+	if (ret)
+		return ret;
+
+	ret = v4l_create_bufs(ops, file, fh, &create);
+	if (ret)
+		return ret;
+
+	ecreate->index = create.index;
+	ecreate->count = create.count;
+	ecreate->capabilities = create.capabilities;
+
+	return 0;
+}
+
+static int v4l_prepare_buf(const struct v4l2_ioctl_ops *ops,
+			   struct file *file, void *fh, void *arg)
+{
+	return v4l_do_buf_op(ops->vidioc_prepare_buf,
+			     ops->vidioc_ext_prepare_buf,
+			     file, fh, arg);
+}
+
+static int v4l_ext_prepare_buf(const struct v4l2_ioctl_ops *ops,
+			       struct file *file, void *fh, void *arg)
+{
+	return v4l_do_ext_buf_op(ops->vidioc_prepare_buf,
+				 ops->vidioc_ext_prepare_buf,
+				 file, fh, arg);
 }
 
 static int v4l_g_parm(const struct v4l2_ioctl_ops *ops,
@@ -3159,6 +3460,86 @@ static int v4l_enum_freq_bands(const struct v4l2_ioctl_ops *ops,
 	return -ENOTTY;
 }
 
+static int v4l_expbuf(const struct v4l2_ioctl_ops *ops, struct file *file,
+		      void *fh, void *arg)
+{
+	struct v4l2_exportbuffer *b = arg;
+	struct v4l2_ext_exportbuffer eb = {
+		.type = b->type,
+		.index = b->index,
+		.first_plane = b->plane,
+		.num_planes = 1,
+		.flags = b->flags,
+	};
+	int ret;
+
+	if (ops->vidioc_expbuf)
+		return ops->vidioc_expbuf(file, fh, b);
+
+	if (b->plane >= VIDEO_MAX_PLANES)
+		return -EINVAL;
+
+	ret = ops->vidioc_ext_expbuf(file, fh, &eb);
+	if (ret)
+		return ret;
+
+	b->fd = eb.fds[b->plane];
+	return 0;
+}
+
+static int v4l_ext_expbuf(const struct v4l2_ioctl_ops *ops,
+			  struct file *file, void *fh, void *arg)
+{
+	struct v4l2_ext_exportbuffer *eb = arg;
+	unsigned int i;
+	int ret;
+
+	if (eb->first_plane >= VIDEO_MAX_PLANES ||
+	    eb->num_planes > VIDEO_MAX_PLANES ||
+	    eb->first_plane + eb->num_planes > VIDEO_MAX_PLANES)
+		return -EINVAL;
+
+	if (ops->vidioc_ext_expbuf)
+		return ops->vidioc_ext_expbuf(file, fh, eb);
+
+	for (i = eb->first_plane; i < eb->first_plane + eb->num_planes; i++) {
+		struct v4l2_exportbuffer b = {
+			.type = eb->type,
+			.index = eb->index,
+			.plane = i,
+			.flags = eb->flags,
+		};
+
+		ret = ops->vidioc_expbuf(file, fh, &b);
+		if (ret)
+			goto err_put_dmabufs;
+
+		eb->fds[i] = b.fd;
+	}
+
+	return 0;
+
+err_put_dmabufs:
+	for (i = eb->first_plane; i < eb->first_plane + eb->num_planes; i++) {
+		struct dma_buf *dmabuf;
+
+		if (eb->fds[i] <= 0)
+			break;
+
+		/*
+		 * We must call dma_buf_put() twice because we got one
+		 * reference taken at dmabuf creation time one taken when
+		 * calling dma_buf_get().
+		 * FIXME: not entirely sure this works correctly.
+		 */
+		dmabuf = dma_buf_get(eb->fds[i]);
+		dma_buf_put(dmabuf);
+		dma_buf_put(dmabuf);
+	}
+
+	return ret;
+}
+
 struct v4l2_ioctl_info {
 	unsigned int ioctl;
 	u32 flags;
@@ -3201,7 +3582,6 @@ struct v4l2_ioctl_info {
 
 DEFINE_V4L_STUB_FUNC(g_fbuf)
 DEFINE_V4L_STUB_FUNC(s_fbuf)
-DEFINE_V4L_STUB_FUNC(expbuf)
 DEFINE_V4L_STUB_FUNC(g_std)
 DEFINE_V4L_STUB_FUNC(g_audio)
 DEFINE_V4L_STUB_FUNC(s_audio)
@@ -3237,12 +3617,16 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
 	IOCTL_INFO(VIDIOC_S_EXT_FMT, v4l_s_ext_fmt, v4l_print_ext_format, INFO_FL_PRIO),
 	IOCTL_INFO(VIDIOC_REQBUFS, v4l_reqbufs, v4l_print_requestbuffers, INFO_FL_PRIO | INFO_FL_QUEUE),
 	IOCTL_INFO(VIDIOC_QUERYBUF, v4l_querybuf, v4l_print_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_buffer, length)),
+	IOCTL_INFO(VIDIOC_EXT_QUERYBUF, v4l_ext_querybuf, v4l_print_ext_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_ext_buffer, num_planes)),
 	IOCTL_INFO(VIDIOC_G_FBUF, v4l_stub_g_fbuf, v4l_print_framebuffer, 0),
 	IOCTL_INFO(VIDIOC_S_FBUF, v4l_stub_s_fbuf, v4l_print_framebuffer, INFO_FL_PRIO),
 	IOCTL_INFO(VIDIOC_OVERLAY, v4l_overlay, v4l_print_u32, INFO_FL_PRIO),
 	IOCTL_INFO(VIDIOC_QBUF, v4l_qbuf, v4l_print_buffer, INFO_FL_QUEUE),
-	IOCTL_INFO(VIDIOC_EXPBUF, v4l_stub_expbuf, v4l_print_exportbuffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_exportbuffer, flags)),
+	IOCTL_INFO(VIDIOC_EXT_QBUF, v4l_ext_qbuf, v4l_print_ext_buffer, INFO_FL_QUEUE),
+	IOCTL_INFO(VIDIOC_EXPBUF, v4l_expbuf, v4l_print_exportbuffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_exportbuffer, flags)),
+	IOCTL_INFO(VIDIOC_EXT_EXPBUF, v4l_ext_expbuf, v4l_print_ext_exportbuffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_exportbuffer, flags)),
 	IOCTL_INFO(VIDIOC_DQBUF, v4l_dqbuf, v4l_print_buffer, INFO_FL_QUEUE),
+	IOCTL_INFO(VIDIOC_EXT_DQBUF, v4l_ext_dqbuf, v4l_print_ext_buffer, INFO_FL_QUEUE),
 	IOCTL_INFO(VIDIOC_STREAMON, v4l_streamon, v4l_print_buftype, INFO_FL_PRIO | INFO_FL_QUEUE),
 	IOCTL_INFO(VIDIOC_STREAMOFF, v4l_streamoff, v4l_print_buftype, INFO_FL_PRIO | INFO_FL_QUEUE),
 	IOCTL_INFO(VIDIOC_G_PARM, v4l_g_parm, v4l_print_streamparm, INFO_FL_CLEAR(v4l2_streamparm, type)),
@@ -3307,7 +3691,9 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
 	IOCTL_INFO(VIDIOC_SUBSCRIBE_EVENT, v4l_subscribe_event, v4l_print_event_subscription, 0),
 	IOCTL_INFO(VIDIOC_UNSUBSCRIBE_EVENT, v4l_unsubscribe_event, v4l_print_event_subscription, 0),
 	IOCTL_INFO(VIDIOC_CREATE_BUFS, v4l_create_bufs, v4l_print_create_buffers, INFO_FL_PRIO | INFO_FL_QUEUE),
+	IOCTL_INFO(VIDIOC_EXT_CREATE_BUFS, v4l_ext_create_bufs, v4l_print_ext_create_buffers, INFO_FL_PRIO | INFO_FL_QUEUE),
 	IOCTL_INFO(VIDIOC_PREPARE_BUF, v4l_prepare_buf, v4l_print_buffer, INFO_FL_QUEUE),
+	IOCTL_INFO(VIDIOC_EXT_PREPARE_BUF, v4l_ext_prepare_buf, v4l_print_ext_buffer, INFO_FL_QUEUE),
 	IOCTL_INFO(VIDIOC_ENUM_DV_TIMINGS, v4l_stub_enum_dv_timings, v4l_print_enum_dv_timings, INFO_FL_CLEAR(v4l2_enum_dv_timings, pad)),
 	IOCTL_INFO(VIDIOC_QUERY_DV_TIMINGS, v4l_stub_query_dv_timings, v4l_print_dv_timings, INFO_FL_ALWAYS_COPY),
 	IOCTL_INFO(VIDIOC_DV_TIMINGS_CAP, v4l_stub_dv_timings_cap, v4l_print_dv_timings_cap, INFO_FL_CLEAR(v4l2_dv_timings_cap, pad)),
include/media/v4l2-ioctl.h +30 -0
index 39ac07fbc7b7..f7e375d38602 100644
@@ -168,16 +168,28 @@ struct v4l2_fh;
  *	:ref:`VIDIOC_REQBUFS <vidioc_reqbufs>` ioctl
  * @vidioc_querybuf: pointer to the function that implements
  *	:ref:`VIDIOC_QUERYBUF <vidioc_querybuf>` ioctl
+ * @vidioc_ext_querybuf: pointer to the function that implements
+ *	:ref:`VIDIOC_EXT_QUERYBUF <vidioc_ext_querybuf>` ioctl
  * @vidioc_qbuf: pointer to the function that implements
  *	:ref:`VIDIOC_QBUF <vidioc_qbuf>` ioctl
+ * @vidioc_ext_qbuf: pointer to the function that implements
+ *	:ref:`VIDIOC_EXT_QBUF <vidioc_ext_qbuf>` ioctl
  * @vidioc_expbuf: pointer to the function that implements
  *	:ref:`VIDIOC_EXPBUF <vidioc_expbuf>` ioctl
+ * @vidioc_ext_expbuf: pointer to the function that implements
+ *	:ref:`VIDIOC_EXT_EXPBUF <vidioc_ext_expbuf>` ioctl
  * @vidioc_dqbuf: pointer to the function that implements
  *	:ref:`VIDIOC_DQBUF <vidioc_qbuf>` ioctl
+ * @vidioc_ext_dqbuf: pointer to the function that implements
+ *	:ref:`VIDIOC_EXT_DQBUF <vidioc_ext_qbuf>` ioctl
  * @vidioc_create_bufs: pointer to the function that implements
  *	:ref:`VIDIOC_CREATE_BUFS <vidioc_create_bufs>` ioctl
+ * @vidioc_ext_create_bufs: pointer to the function that implements
+ *	:ref:`VIDIOC_EXT_CREATE_BUFS <vidioc_ext_create_bufs>` ioctl
  * @vidioc_prepare_buf: pointer to the function that implements
  *	:ref:`VIDIOC_PREPARE_BUF <vidioc_prepare_buf>` ioctl
+ * @vidioc_ext_prepare_buf: pointer to the function that implements
+ *	:ref:`VIDIOC_EXT_PREPARE_BUF <vidioc_ext_prepare_buf>` ioctl
  * @vidioc_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_OVERLAY <vidioc_overlay>` ioctl
  * @vidioc_g_fbuf: pointer to the function that implements
@@ -438,17 +450,29 @@ struct v4l2_ioctl_ops {
 			      struct v4l2_requestbuffers *b);
 	int (*vidioc_querybuf)(struct file *file, void *fh,
 			       struct v4l2_buffer *b);
+	int (*vidioc_ext_querybuf)(struct file *file, void *fh,
+				   struct v4l2_ext_buffer *b);
 	int (*vidioc_qbuf)(struct file *file, void *fh,
 			   struct v4l2_buffer *b);
+	int (*vidioc_ext_qbuf)(struct file *file, void *fh,
+			       struct v4l2_ext_buffer *b);
 	int (*vidioc_expbuf)(struct file *file, void *fh,
 			     struct v4l2_exportbuffer *e);
+	int (*vidioc_ext_expbuf)(struct file *file, void *fh,
+				 struct v4l2_ext_exportbuffer *e);
 	int (*vidioc_dqbuf)(struct file *file, void *fh,
 			    struct v4l2_buffer *b);
+	int (*vidioc_ext_dqbuf)(struct file *file, void *fh,
+				struct v4l2_ext_buffer *b);
 
 	int (*vidioc_create_bufs)(struct file *file, void *fh,
 				  struct v4l2_create_buffers *b);
+	int (*vidioc_ext_create_bufs)(struct file *file, void *fh,
+				      struct v4l2_ext_create_buffers *b);
 	int (*vidioc_prepare_buf)(struct file *file, void *fh,
 				  struct v4l2_buffer *b);
+	int (*vidioc_ext_prepare_buf)(struct file *file, void *fh,
+				      struct v4l2_ext_buffer *b);
 
 	int (*vidioc_overlay)(struct file *file, void *fh, unsigned int i);
 	int (*vidioc_g_fbuf)(struct file *file, void *fh,
@@ -757,4 +781,10 @@ int v4l2_ext_format_to_format(const struct v4l2_ext_format *e,
 			      struct v4l2_format *f,
 			      bool mplane_cap, bool strict);
 
+int v4l2_ext_buffer_to_buffer(const struct v4l2_ext_buffer *e,
+			      struct v4l2_buffer *b,
+			      bool mplane_cap);
+int v4l2_buffer_to_ext_buffer(const struct v4l2_buffer *b,
+			      struct v4l2_ext_buffer *e);
+
 #endif /* _V4L2_IOCTL_H */
include/uapi/linux/videodev2.h +130 -0
index c7b169de1c7b..33c8348df13f 100644
@@ -953,6 +953,49 @@ struct v4l2_plane {
 	__u32			reserved[11];
 };
 
+/**
+ * struct v4l2_ext_plane - extended plane buffer info
+ * @bytesused: number of bytes occupied by data in the plane (payload)
+ * @length: size of this plane (NOT the payload) in bytes
+ * @mem_offset: when memory in the associated struct v4l2_ext_buffer is
+ *		V4L2_MEMORY_MMAP, equals the offset from the start of the
+ *		device memory for this plane (or is a "cookie" that should be
+ *		passed to mmap() called on the video node)
+ * @userptr: when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
+ *	     to this plane
+ * @dmabuf.fd: when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
+ *	       associated with this plane
+ * @dmabuf.offset: where the plane starts inside the DMABUF buffer. All planes
+ *		   might share the same buffer object. In this case we need to
+ *		   know where the plane start inside this buffer.
+ * @data_offset: offset in the plane to the start of data; usually 0, unless
+ *		 there is a header in front of the data. data_offset is
+ *		 relative to start_offset, so absolute data_offset is actually
+ *		 start_offset + data_offset
+ *
+ *
+ * Multi-planar buffers consist of one or more planes, e.g. an YCbCr buffer
+ * with two planes can have one plane for Y, and another for interleaved CbCr
+ * components. Each plane can reside in a separate memory buffer, or even in
+ * a completely separate memory node (e.g. in embedded devices).
+ * Note that this struct is also used for uni-planar buffers, but in that case
+ * you'll only have one plane defined.
+ */
+struct v4l2_ext_plane {
+	__u32 bytesused;
+	__u32 length;
+	union {
+		__u32 mem_offset;
+		__u64 userptr;
+		struct {
+			__s32 fd;
+			__u32 offset;
+		} dmabuf;
+	} m;
+	__u32 data_offset;
+	__u32 reserved[10];
+};
+
 /**
  * struct v4l2_buffer - video buffer info
  * @index:	id number of the buffer
@@ -1010,6 +1053,40 @@ struct v4l2_buffer {
 	};
 };
 
+/**
+ * struct v4l2_ext_buffer - extended video buffer info
+ * @index: id number of the buffer
+ * @type: enum v4l2_buf_type; buffer type. _MPLANE and _OVERLAY formats are
+ *	  invalid
+ * @flags: buffer informational flags
+ * @field: enum v4l2_field; field order of the image in the buffer
+ * @timestamp: frame timestamp
+ * @sequence: sequence count of this frame
+ * @memory: enum v4l2_memory; the method, in which the actual video data is
+ *          passed
+ * @planes: per-plane buffer information
+ * @num_planes: number of plane buffers
+ * @request_fd: fd of the request that this buffer should use
+ * @reserved: some extra space reserved to add future fields (like timecode).
+ *	      Must be set to 0
+ *
+ * Contains data exchanged by application and driver using one of the Streaming
+ * I/O methods.
+ */
+struct v4l2_ext_buffer {
+	__u32 index;
+	__u32 type;
+	__u32 flags;
+	__u32 field;
+	__u64 timestamp;
+	__u32 sequence;
+	__u32 memory;
+	struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
+	__u32 num_planes;
+	__u32 request_fd;
+	__u32 reserved[10];
+};
+
 /**
  * v4l2_timeval_to_ns - Convert timeval to nanoseconds
  * @ts:		pointer to the timeval variable to be converted
@@ -1087,6 +1164,35 @@ struct v4l2_exportbuffer {
 	__u32		reserved[11];
 };
 
+/**
+ * struct v4l2_ext_exportbuffer - export of video buffer as DMABUF file
+ *				  descriptor using extended format
+ *
+ * @index: id number of the buffer
+ * @type: enum v4l2_buf_type; buffer type
+ * @flags: flags for newly created file(s), currently only O_CLOEXEC is
+ *	   supported, refer to manual of open syscall for more details
+ * @first_plane: first plane to export. Most likely set to 0
+ * @num_planes: number of planes to export. Most set to the number of planes
+ *		attached to the buffer
+ * @fds: file descriptors associated with DMABUF (set by driver). Note that all
+ *	 planes might share the same buffer and then be returned the same FD
+ *
+ * Contains data used for exporting a video buffer as DMABUF file descriptor.
+ * The buffer is identified by a 'cookie' returned by VIDIOC_QUERYBUF
+ * (identical to the cookie used to mmap() the buffer to userspace). All
+ * reserved fields must be set to zero.
+ */
+struct v4l2_ext_exportbuffer {
+	__u32 type; /* enum v4l2_buf_type */
+	__u32 index;
+	__u32 flags;
+	__u32 first_plane;
+	__u32 num_planes;
+	__s32 fds[VIDEO_MAX_PLANES];
+	__u32 reserved;
+};
+
 /*
  *	O V E R L A Y   P R E V I E W
  */
@@ -2483,6 +2589,23 @@ struct v4l2_create_buffers {
 	__u32			reserved[7];
 };
 
+/**
+ * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
+ * @index:	on return, index of the first created buffer
+ * @count:	entry: number of requested buffers,
+ *		return: number of created buffers
+ * @memory:	enum v4l2_memory; buffer memory type
+ * @capabilities: capabilities of this buffer type.
+ * @format:	frame format, for which buffers are requested
+ */
+struct v4l2_ext_create_buffers {
+	__u32			index;
+	__u32			count;
+	__u32			memory;
+	__u32			capabilities;
+	struct v4l2_ext_format	format;
+};
+
 /*
  *	I O C T L   C O D E S   F O R   V I D E O   D E V I C E S
  *
@@ -2586,6 +2709,13 @@ struct v4l2_create_buffers {
 #define VIDIOC_G_EXT_FMT	_IOWR('V', 104, struct v4l2_ext_format)
 #define VIDIOC_S_EXT_FMT	_IOWR('V', 105, struct v4l2_ext_format)
 #define VIDIOC_TRY_EXT_FMT	_IOWR('V', 106, struct v4l2_ext_format)
+#define VIDIOC_EXT_CREATE_BUFS	_IOWR('V', 107, struct v4l2_ext_create_buffers)
+#define VIDIOC_EXT_QUERYBUF	_IOWR('V', 108, struct v4l2_ext_buffer)
+#define VIDIOC_EXT_QBUF		_IOWR('V', 109, struct v4l2_ext_buffer)
+#define VIDIOC_EXT_DQBUF	_IOWR('V', 110, struct v4l2_ext_buffer)
+#define VIDIOC_EXT_PREPARE_BUF	_IOWR('V', 111, struct v4l2_ext_buffer)
+#define VIDIOC_EXT_EXPBUF	_IOWR('V', 112, struct v4l2_ext_exportbuffer)
+
 /* Reminder: when adding new ioctls please add support for them to
    drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
 
Signature · 1 line
2.21.0
#12 lore raw

Re: [RFC PATCH 3/3] media: v4l2: Add extended buffer operations

Quoted 348 lines · Fritz Koenig

Thanks for putting this patch set together. I'm currently trying to get modifiers working for UBWC. I've applied these patches and with a little tweaking can get video decode working without modifiers. I haven't tried the new ioctls yet.

Quoted 144 lines

From: Hans Verkuil <[email protected]>

Those extended buffer ops have several purpose:
1/ Fix y2038 issues by converting the timestamp into an u64 counting
the number of ns elapsed since 1970
2/ Unify single/multiplanar handling
3/ Add a new start offset field to each v4l2 plane buffer info struct
to support the case where a single buffer object is storing all planes data, each one being placed at a different offset

New hooks are created in v4l2_ioctl_ops so that drivers can start using these new objects.

The core takes care of converting new ioctls requests to old ones if the driver does not support the new hooks, and vice versa.

Note that the timecode field is gone, since there doesn't seem to be in-kernel users, but can be added back in the reserved area if needed.

---
Changes in v3:
- Rebased on top of media/master (post 5.4-rc1)

Changes in v2:
- Add reserved space to v4l2_ext_buffer so that new fields can be added later on

---
 drivers/media/v4l2-core/v4l2-dev.c   |  30 +-
 drivers/media/v4l2-core/v4l2-ioctl.c | 428 +++++++++++++++++++++++++--
 include/media/v4l2-ioctl.h           |  30 ++
 include/uapi/linux/videodev2.h       | 130 ++++++++
 4 files changed, 591 insertions(+), 27 deletions(-)
drivers/media/v4l2-core/v4l2-dev.c +24 -6
index 9aad715537b3..35c8caccd025 100644
@@ -696,12 +696,30 @@ static void determine_valid_ioctls(struct video_device *vdev)
      if (is_vid || is_vbi || is_sdr || is_tch) {
              /* ioctls valid for video, metadata, vbi or sdr */
              SET_VALID_IOCTL(ops, VIDIOC_REQBUFS, vidioc_reqbufs);
-             SET_VALID_IOCTL(ops, VIDIOC_QUERYBUF, vidioc_querybuf);
-             SET_VALID_IOCTL(ops, VIDIOC_QBUF, vidioc_qbuf);
-             SET_VALID_IOCTL(ops, VIDIOC_EXPBUF, vidioc_expbuf);
-             SET_VALID_IOCTL(ops, VIDIOC_DQBUF, vidioc_dqbuf);
-             SET_VALID_IOCTL(ops, VIDIOC_CREATE_BUFS, vidioc_create_bufs);
-             SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, vidioc_prepare_buf);
+             if (ops->vidioc_querybuf || ops->vidioc_ext_querybuf) {
+                     set_bit(_IOC_NR(VIDIOC_QUERYBUF), valid_ioctls);
+                     set_bit(_IOC_NR(VIDIOC_EXT_QUERYBUF), valid_ioctls);
+             }
+             if (ops->vidioc_qbuf || ops->vidioc_ext_qbuf) {
+                     set_bit(_IOC_NR(VIDIOC_QBUF), valid_ioctls);
+                     set_bit(_IOC_NR(VIDIOC_EXT_QBUF), valid_ioctls);
+             }
+             if (ops->vidioc_expbuf || ops->vidioc_ext_expbuf) {
+                     set_bit(_IOC_NR(VIDIOC_EXPBUF), valid_ioctls);
+                     set_bit(_IOC_NR(VIDIOC_EXT_EXPBUF), valid_ioctls);
+             }
+             if (ops->vidioc_dqbuf || ops->vidioc_ext_dqbuf) {
+                     set_bit(_IOC_NR(VIDIOC_DQBUF), valid_ioctls);
+                     set_bit(_IOC_NR(VIDIOC_EXT_DQBUF), valid_ioctls);
+             }
+             if (ops->vidioc_create_bufs || ops->vidioc_ext_create_bufs) {
+                     set_bit(_IOC_NR(VIDIOC_CREATE_BUFS), valid_ioctls);
+                     set_bit(_IOC_NR(VIDIOC_EXT_CREATE_BUFS), valid_ioctls);
+             }
+             if (ops->vidioc_prepare_buf || ops->vidioc_ext_prepare_buf) {
+                     set_bit(_IOC_NR(VIDIOC_PREPARE_BUF), valid_ioctls);
+                     set_bit(_IOC_NR(VIDIOC_EXT_PREPARE_BUF), valid_ioctls);
+             }
              SET_VALID_IOCTL(ops, VIDIOC_STREAMON, vidioc_streamon);
              SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, vidioc_streamoff);
      }
drivers/media/v4l2-core/v4l2-ioctl.c +40 -0
index 78e14c1dc76f..356218e44ccb 100644
@@ -579,6 +579,25 @@ static void v4l_print_buffer(const void *arg, bool write_only)
                      tc->type, tc->flags, tc->frames, *(__u32 *)tc->userbits);
 }
 
+static void v4l_print_ext_buffer(const void *arg, bool write_only)
+{
+     const struct v4l2_ext_buffer *p = arg;
+     const struct v4l2_ext_plane *plane;
+     int i;
+
+     pr_cont("%lld index=%d, type=%s, flags=0x%08x, field=%s, sequence=%d, memory=%s\n",
+             p->timestamp, p->index, prt_names(p->type, v4l2_type_names),
+             p->flags, prt_names(p->field, v4l2_field_names),
+             p->sequence, prt_names(p->memory, v4l2_memory_names));
+
+     for (i = 0; i < p->num_planes; ++i) {
+             plane = &p->planes[i];
+             pr_debug("plane %d: bytesused=%d, data_offset=0x%08x, offset/userptr=0x%llx, length=%d\n",
+                      i, plane->bytesused, plane->data_offset,
+                      plane->m.userptr, plane->length);
+     }
+}
+
 static void v4l_print_exportbuffer(const void *arg, bool write_only)
 {
      const struct v4l2_exportbuffer *p = arg;
@@ -588,6 +607,18 @@ static void v4l_print_exportbuffer(const void *arg, bool write_only)
              p->index, p->plane, p->flags);
 }
 
+static void v4l_print_ext_exportbuffer(const void *arg, bool write_only)
+{
+     const struct v4l2_ext_exportbuffer *p = arg;
+     unsigned int i;
+
+     pr_cont("type=%s, index=%u, first_plane=%u num_planes=%u, flags=%08x\n",
+             prt_names(p->type, v4l2_type_names), p->index, p->first_plane,
+             p->num_planes, p->flags);
+     for (i = p->first_plane; i < p->first_plane + p->num_planes; ++i)
+             pr_debug("plane %u: fd=%d\n", i, p->fds[i]);
+}
+
 static void v4l_print_create_buffers(const void *arg, bool write_only)
 {
      const struct v4l2_create_buffers *p = arg;
@@ -598,6 +629,15 @@ static void v4l_print_create_buffers(const void *arg, bool write_only)
      v4l_print_format(&p->format, write_only);
 }
 
+static void v4l_print_ext_create_buffers(const void *arg, bool write_only)
+{
+     const struct v4l2_ext_create_buffers *p = arg;
+
+     pr_cont("index=%d, count=%d, memory=%s, ", p->index, p->count,
+             prt_names(p->memory, v4l2_memory_names));
+     v4l_print_ext_format(&p->format, write_only);
+}
+
 static void v4l_print_streamparm(const void *arg, bool write_only)
 {
      const struct v4l2_streamparm *p = arg;
@@ -1319,6 +1359,123 @@ int v4l2_format_to_ext_format(const struct v4l2_format *f,
 }
 EXPORT_SYMBOL_GPL(v4l2_format_to_ext_format);

drivers/media/v4l2-core/v4l2-ioctl.c int v4l2_format_to_ext_format(const struct v4l2_format *f, · line 1370

Quoted 10 lines
drivers/media/v4l2-core/v4l2-ioctl.c +9 -0
 
+int v4l2_ext_buffer_to_buffer(const struct v4l2_ext_buffer *e,
+                           struct v4l2_buffer *b, bool mplane_cap)
+{
+     u64 nsecs;
+
+     if (!mplane_cap && e->num_planes > 1)
+             return -EINVAL;
+
+     memset(b, 0, sizeof(*b));

b->planes needs to be preserved.

I was not able to find a tree that these patches applied to cleanly. In v5.4 the struct v4l2_buffer member planes is a raw pointer, so this memset will clear it out.

Quoted 182 lines
drivers/media/v4l2-core/v4l2-ioctl.c +168 -5
+
+     b->index = e->index;
+     b->flags = e->flags;
+     b->field = e->field;
+     b->sequence = e->sequence;
+     b->memory = e->memory;
+     b->request_fd = e->request_fd;
+     b->timestamp.tv_sec = div64_u64_rem(e->timestamp, NSEC_PER_SEC, &nsecs);
+     b->timestamp.tv_usec = (u32)nsecs / NSEC_PER_USEC;
+     if (mplane_cap) {
+             unsigned int i;
+
+             if (e->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
+                     b->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
+             else
+                     b->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
+
+             b->length = e->num_planes;
+             for (i = 0; i < e->num_planes; i++) {
+                     if (b->memory == V4L2_MEMORY_DMABUF) {
+                             if (e->planes[i].m.dmabuf.offset)
+                                     return -EINVAL;
+
+                             b->m.planes[i].m.fd = e->planes[i].m.dmabuf.fd;
+                     } else {
+                             b->m.planes[i].m.userptr = e->planes[i].m.userptr;
+                     }
+                     b->m.planes[i].length = e->planes[i].length;
+                     b->m.planes[i].bytesused = e->planes[i].bytesused;
+                     b->m.planes[i].data_offset = e->planes[i].data_offset;
+                     memset(b->m.planes[i].reserved, 0,
+                            sizeof(b->m.planes[i].reserved));
+             }
+     } else {
+             b->type = e->type;
+             b->bytesused = e->planes[0].bytesused;
+             b->length = e->planes[0].length;
+             if (b->memory == V4L2_MEMORY_DMABUF) {
+                     if (e->planes[0].m.dmabuf.offset)
+                             return -EINVAL;
+
+                     b->m.fd = e->planes[0].m.dmabuf.fd;
+             } else {
+                     b->m.userptr = e->planes[0].m.userptr;
+             }
+     }
+
+     return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_ext_buffer_to_buffer);
+
+int v4l2_buffer_to_ext_buffer(const struct v4l2_buffer *b,
+                           struct v4l2_ext_buffer *e)
+{
+     memset(e, 0, sizeof(*e));
+
+     e->index = b->index;
+     e->flags = b->flags;
+     e->field = b->field;
+     e->sequence = b->sequence;
+     e->memory = b->memory;
+     e->request_fd = b->request_fd;
+     e->timestamp = b->timestamp.tv_sec * NSEC_PER_SEC +
+             b->timestamp.tv_usec * NSEC_PER_USEC;
+     if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
+             unsigned int i;
+
+             if (!b->m.planes)
+                     return -EINVAL;
+
+             if (b->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
+                     e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+             else
+                     e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
+
+             e->num_planes = b->length;
+             for (i = 0; i < e->num_planes; i++) {
+                     if (b->memory == V4L2_MEMORY_DMABUF) {
+                             e->planes[i].m.dmabuf.fd = b->m.planes[i].m.fd;
+                             e->planes[i].m.dmabuf.offset = 0;
+                     } else {
+                             e->planes[i].m.userptr = b->m.planes[i].m.userptr;
+                     }
+                     e->planes[i].length = b->m.planes[i].length;
+                     e->planes[i].bytesused = b->m.planes[i].bytesused;
+                     e->planes[i].data_offset = b->m.planes[i].data_offset;
+                     memset(e->planes[i].reserved, 0,
+                            sizeof(e->planes[i].reserved));
+             }
+     } else {
+             e->type = b->type;
+             e->num_planes = 1;
+             e->planes[0].bytesused = b->bytesused;
+             e->planes[0].length = b->length;
+             if (b->memory == V4L2_MEMORY_DMABUF) {
+                     e->planes[0].m.dmabuf.fd = b->m.fd;
+                     e->planes[0].m.dmabuf.offset = 0;
+             } else {
+                     e->planes[0].m.userptr = b->m.userptr;
+             }
+             e->planes[0].m.userptr = b->m.userptr;
+             e->planes[0].data_offset = 0;
+     }
+
+     return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_buffer_to_ext_buffer);
+
 static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
                              struct file *file, void *fh, void *arg)
 {
@@ -2506,31 +2663,109 @@ static int v4l_reqbufs(const struct v4l2_ioctl_ops *ops,
      return ops->vidioc_reqbufs(file, fh, p);
 }
 
-static int v4l_querybuf(const struct v4l2_ioctl_ops *ops,
-                             struct file *file, void *fh, void *arg)
+static int v4l_do_buf_op(int (*op)(struct file *, void *,
+                                struct v4l2_buffer *),
+                      int (*ext_op)(struct file *, void *,
+                                    struct v4l2_ext_buffer *),
+                      struct file *file, void *fh, struct v4l2_buffer *b)
 {
-     struct v4l2_buffer *p = arg;
-     int ret = check_fmt(file, p->type);
+     struct v4l2_ext_buffer eb;
+     int ret;
 
-     return ret ? ret : ops->vidioc_querybuf(file, fh, p);
+     ret = check_fmt(file, b->type);
+     if (ret)
+             return ret;
+
+     if (op)
+             return op(file, fh, b);
+
+     ret = v4l2_buffer_to_ext_buffer(b, &eb);
+     if (ret)
+             return ret;
+
+     ret = ext_op(file, fh, &eb);
+     if (ret)
+             return ret;
+
+     v4l2_ext_buffer_to_buffer(&eb, b, V4L2_TYPE_IS_MULTIPLANAR(b->type));
+     return 0;
+}
+
+static int v4l_do_ext_buf_op(int (*op)(struct file *, void *,
+                                    struct v4l2_buffer *),
+                          int (*ext_op)(struct file *, void *,
+                                        struct v4l2_ext_buffer *),
+                          struct file *file, void *fh,
+                          struct v4l2_ext_buffer *eb)
+{
+     struct video_device *vdev = video_devdata(file);
+     struct v4l2_buffer b;
+     bool mplane_cap;
+     int ret;
+
+     ret = check_fmt(file, eb->type);
+     if (ret)
+             return ret;
+
+     if (ext_op)
+             return ext_op(file, fh, eb);
+
+     mplane_cap = !!(vdev->device_caps &
+                     (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
+                      V4L2_CAP_VIDEO_OUTPUT_MPLANE |
+                      V4L2_CAP_VIDEO_M2M_MPLANE));
+     ret = v4l2_ext_buffer_to_buffer(eb, &b, mplane_cap);
+     if (ret)
+             return ret;
+
+     ret = op(file, fh, &b);
+     if (ret)
+             return ret;
+
+     v4l2_buffer_to_ext_buffer(&b, eb);
+     return 0;
+}
Quoted 8 lines

drivers/media/v4l2-core/v4l2-ioctl.c static int v4l_reqbufs(const struct v4l2_ioctl_ops *ops, · line 2735

Quoted 8 lines
drivers/media/v4l2-core/v4l2-ioctl.c +8 -0
+
+static int v4l_querybuf(const struct v4l2_ioctl_ops *ops,
+                     struct file *file, void *fh, void *arg)
+{
+     return v4l_do_buf_op(ops->vidioc_querybuf, ops->vidioc_ext_querybuf,
+                          file, fh, arg);
+}
+

VIDIOC_QUERYBUF has a problem now.
v4l_do_buf_op macro expands into:
v4l2_buffer_to_ext_buffer
op
v4l2_ext_buffer_to_buffer

To copy to an v4l2_ext_buffer, type is changed by v4l2_buffer_to_ext_buffer from V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE to V4L2_BUF_TYPE_VIDEO_CAPTURE.

When __fill_v4l2_buffer is called from vb2_core_querybuf, type is set back to the V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE.

Finally when v4l2_ext_buffer_to_buffer is called the queue ends up getting set to V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE.

The reason I see this is that my OUTPUT queue is mm and my CAPTURE queue is dmabuf. When I try to then VIDIOC_QBUF that buffer, the memory types for the queue and the buffer do not match.

Quoted 508 lines
Quoted 508 lines
drivers/media/v4l2-core/v4l2-ioctl.c +182 -16
+static int v4l_ext_querybuf(const struct v4l2_ioctl_ops *ops,
+                         struct file *file, void *fh, void *arg)
+{
+     return v4l_do_ext_buf_op(ops->vidioc_querybuf,
+                              ops->vidioc_ext_querybuf, file, fh, arg);
 }
 
 static int v4l_qbuf(const struct v4l2_ioctl_ops *ops,
-                             struct file *file, void *fh, void *arg)
+                 struct file *file, void *fh, void *arg)
 {
-     struct v4l2_buffer *p = arg;
-     int ret = check_fmt(file, p->type);
+     return v4l_do_buf_op(ops->vidioc_qbuf, ops->vidioc_ext_qbuf,
+                          file, fh, arg);
+}
 
-     return ret ? ret : ops->vidioc_qbuf(file, fh, p);
+static int v4l_ext_qbuf(const struct v4l2_ioctl_ops *ops,
+                     struct file *file, void *fh, void *arg)
+{
+     return v4l_do_ext_buf_op(ops->vidioc_qbuf, ops->vidioc_ext_qbuf,
+                              file, fh, arg);
 }
 
 static int v4l_dqbuf(const struct v4l2_ioctl_ops *ops,
-                             struct file *file, void *fh, void *arg)
+                  struct file *file, void *fh, void *arg)
 {
-     struct v4l2_buffer *p = arg;
-     int ret = check_fmt(file, p->type);
+     return v4l_do_buf_op(ops->vidioc_dqbuf, ops->vidioc_ext_dqbuf,
+                          file, fh, arg);
+}
 
-     return ret ? ret : ops->vidioc_dqbuf(file, fh, p);
+static int v4l_ext_dqbuf(const struct v4l2_ioctl_ops *ops,
+                      struct file *file, void *fh, void *arg)
+{
+     return v4l_do_ext_buf_op(ops->vidioc_dqbuf, ops->vidioc_ext_dqbuf,
+                              file, fh, arg);
 }
 
 static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
@@ -2546,7 +2781,27 @@ static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
 
      v4l_sanitize_format(&create->format);
 
-     ret = ops->vidioc_create_bufs(file, fh, create);
+     if (ops->vidioc_create_bufs) {
+             ret = ops->vidioc_create_bufs(file, fh, create);
+     } else {
+             struct v4l2_ext_create_buffers ecreate = {
+                     .count = create->count,
+                     .memory = create->memory,
+             };
+
+             ret = v4l2_format_to_ext_format(&create->format,
+                                             &ecreate.format, true);
+             if (ret)
+                     return ret;
+
+             ret = ops->vidioc_ext_create_bufs(file, fh, &ecreate);
+             if (ret)
+                     return ret;
+
+             create->index = ecreate.index;
+             create->count = ecreate.count;
+             create->capabilities = ecreate.capabilities;
+     }
 
      if (create->format.type == V4L2_BUF_TYPE_VIDEO_CAPTURE ||
          create->format.type == V4L2_BUF_TYPE_VIDEO_OUTPUT)
@@ -2555,13 +2810,59 @@ static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
      return ret;
 }
 
-static int v4l_prepare_buf(const struct v4l2_ioctl_ops *ops,
-                             struct file *file, void *fh, void *arg)
+static int v4l_ext_create_bufs(const struct v4l2_ioctl_ops *ops,
+                            struct file *file, void *fh, void *arg)
 {
-     struct v4l2_buffer *b = arg;
-     int ret = check_fmt(file, b->type);
+     struct v4l2_ext_create_buffers *ecreate = arg;
+     struct video_device *vdev = video_devdata(file);
+     struct v4l2_create_buffers create = {
+             .count = ecreate->count,
+             .memory = ecreate->memory,
+     };
+     bool mplane_cap;
+     int ret;
 
-     return ret ? ret : ops->vidioc_prepare_buf(file, fh, b);
+     ret = check_fmt(file, ecreate->format.type);
+     if (ret)
+             return ret;
+
+     if (ops->vidioc_ext_create_bufs)
+             return ops->vidioc_ext_create_bufs(file, fh, ecreate);
+
+     mplane_cap = !!(vdev->device_caps &
+                     (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
+                      V4L2_CAP_VIDEO_OUTPUT_MPLANE |
+                      V4L2_CAP_VIDEO_M2M_MPLANE));
+     ret = v4l2_ext_format_to_format(&ecreate->format,
+                                     &create.format, mplane_cap, true);
+     if (ret)
+             return ret;
+
+     ret = v4l_create_bufs(ops, file, fh, &create);
+     if (ret)
+             return ret;
+
+     ecreate->index = create.index;
+     ecreate->count = create.count;
+     ecreate->capabilities = create.capabilities;
+
+     return 0;
+}
+
+static int v4l_prepare_buf(const struct v4l2_ioctl_ops *ops,
+                        struct file *file, void *fh, void *arg)
+{
+     return v4l_do_buf_op(ops->vidioc_prepare_buf,
+                          ops->vidioc_ext_prepare_buf,
+                          file, fh, arg);
+}
+
+static int v4l_ext_prepare_buf(const struct v4l2_ioctl_ops *ops,
+                            struct file *file, void *fh, void *arg)
+{
+     return v4l_do_ext_buf_op(ops->vidioc_prepare_buf,
+                              ops->vidioc_ext_prepare_buf,
+                              file, fh, arg);
 }
 
 static int v4l_g_parm(const struct v4l2_ioctl_ops *ops,
@@ -3159,6 +3460,86 @@ static int v4l_enum_freq_bands(const struct v4l2_ioctl_ops *ops,
      return -ENOTTY;
 }
 
+static int v4l_expbuf(const struct v4l2_ioctl_ops *ops, struct file *file,
+                   void *fh, void *arg)
+{
+     struct v4l2_exportbuffer *b = arg;
+     struct v4l2_ext_exportbuffer eb = {
+             .type = b->type,
+             .index = b->index,
+             .first_plane = b->plane,
+             .num_planes = 1,
+             .flags = b->flags,
+     };
+     int ret;
+
+     if (ops->vidioc_expbuf)
+             return ops->vidioc_expbuf(file, fh, b);
+
+     if (b->plane >= VIDEO_MAX_PLANES)
+             return -EINVAL;
+
+     ret = ops->vidioc_ext_expbuf(file, fh, &eb);
+     if (ret)
+             return ret;
+
+     b->fd = eb.fds[b->plane];
+     return 0;
+}
+
+static int v4l_ext_expbuf(const struct v4l2_ioctl_ops *ops,
+                       struct file *file, void *fh, void *arg)
+{
+     struct v4l2_ext_exportbuffer *eb = arg;
+     unsigned int i;
+     int ret;
+
+     if (eb->first_plane >= VIDEO_MAX_PLANES ||
+         eb->num_planes > VIDEO_MAX_PLANES ||
+         eb->first_plane + eb->num_planes > VIDEO_MAX_PLANES)
+             return -EINVAL;
+
+     if (ops->vidioc_ext_expbuf)
+             return ops->vidioc_ext_expbuf(file, fh, eb);
+
+     for (i = eb->first_plane; i < eb->first_plane + eb->num_planes; i++) {
+             struct v4l2_exportbuffer b = {
+                     .type = eb->type,
+                     .index = eb->index,
+                     .plane = i,
+                     .flags = eb->flags,
+             };
+
+             ret = ops->vidioc_expbuf(file, fh, &b);
+             if (ret)
+                     goto err_put_dmabufs;
+
+             eb->fds[i] = b.fd;
+     }
+
+     return 0;
+
+err_put_dmabufs:
+     for (i = eb->first_plane; i < eb->first_plane + eb->num_planes; i++) {
+             struct dma_buf *dmabuf;
+
+             if (eb->fds[i] <= 0)
+                     break;
+
+             /*
+              * We must call dma_buf_put() twice because we got one
+              * reference taken at dmabuf creation time one taken when
+              * calling dma_buf_get().
+              * FIXME: not entirely sure this works correctly.
+              */
+             dmabuf = dma_buf_get(eb->fds[i]);
+             dma_buf_put(dmabuf);
+             dma_buf_put(dmabuf);
+     }
+
+     return ret;
+}
+
 struct v4l2_ioctl_info {
      unsigned int ioctl;
      u32 flags;
@@ -3201,7 +3582,6 @@ struct v4l2_ioctl_info {
 
 DEFINE_V4L_STUB_FUNC(g_fbuf)
 DEFINE_V4L_STUB_FUNC(s_fbuf)
-DEFINE_V4L_STUB_FUNC(expbuf)
 DEFINE_V4L_STUB_FUNC(g_std)
 DEFINE_V4L_STUB_FUNC(g_audio)
 DEFINE_V4L_STUB_FUNC(s_audio)
@@ -3237,12 +3617,16 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
      IOCTL_INFO(VIDIOC_S_EXT_FMT, v4l_s_ext_fmt, v4l_print_ext_format, INFO_FL_PRIO),
      IOCTL_INFO(VIDIOC_REQBUFS, v4l_reqbufs, v4l_print_requestbuffers, INFO_FL_PRIO | INFO_FL_QUEUE),
      IOCTL_INFO(VIDIOC_QUERYBUF, v4l_querybuf, v4l_print_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_buffer, length)),
+     IOCTL_INFO(VIDIOC_EXT_QUERYBUF, v4l_ext_querybuf, v4l_print_ext_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_ext_buffer, num_planes)),
      IOCTL_INFO(VIDIOC_G_FBUF, v4l_stub_g_fbuf, v4l_print_framebuffer, 0),
      IOCTL_INFO(VIDIOC_S_FBUF, v4l_stub_s_fbuf, v4l_print_framebuffer, INFO_FL_PRIO),
      IOCTL_INFO(VIDIOC_OVERLAY, v4l_overlay, v4l_print_u32, INFO_FL_PRIO),
      IOCTL_INFO(VIDIOC_QBUF, v4l_qbuf, v4l_print_buffer, INFO_FL_QUEUE),
-     IOCTL_INFO(VIDIOC_EXPBUF, v4l_stub_expbuf, v4l_print_exportbuffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_exportbuffer, flags)),
+     IOCTL_INFO(VIDIOC_EXT_QBUF, v4l_ext_qbuf, v4l_print_ext_buffer, INFO_FL_QUEUE),
+     IOCTL_INFO(VIDIOC_EXPBUF, v4l_expbuf, v4l_print_exportbuffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_exportbuffer, flags)),
+     IOCTL_INFO(VIDIOC_EXT_EXPBUF, v4l_ext_expbuf, v4l_print_ext_exportbuffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_exportbuffer, flags)),
      IOCTL_INFO(VIDIOC_DQBUF, v4l_dqbuf, v4l_print_buffer, INFO_FL_QUEUE),
+     IOCTL_INFO(VIDIOC_EXT_DQBUF, v4l_ext_dqbuf, v4l_print_ext_buffer, INFO_FL_QUEUE),
      IOCTL_INFO(VIDIOC_STREAMON, v4l_streamon, v4l_print_buftype, INFO_FL_PRIO | INFO_FL_QUEUE),
      IOCTL_INFO(VIDIOC_STREAMOFF, v4l_streamoff, v4l_print_buftype, INFO_FL_PRIO | INFO_FL_QUEUE),
      IOCTL_INFO(VIDIOC_G_PARM, v4l_g_parm, v4l_print_streamparm, INFO_FL_CLEAR(v4l2_streamparm, type)),
@@ -3307,7 +3691,9 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
      IOCTL_INFO(VIDIOC_SUBSCRIBE_EVENT, v4l_subscribe_event, v4l_print_event_subscription, 0),
      IOCTL_INFO(VIDIOC_UNSUBSCRIBE_EVENT, v4l_unsubscribe_event, v4l_print_event_subscription, 0),
      IOCTL_INFO(VIDIOC_CREATE_BUFS, v4l_create_bufs, v4l_print_create_buffers, INFO_FL_PRIO | INFO_FL_QUEUE),
+     IOCTL_INFO(VIDIOC_EXT_CREATE_BUFS, v4l_ext_create_bufs, v4l_print_ext_create_buffers, INFO_FL_PRIO | INFO_FL_QUEUE),
      IOCTL_INFO(VIDIOC_PREPARE_BUF, v4l_prepare_buf, v4l_print_buffer, INFO_FL_QUEUE),
+     IOCTL_INFO(VIDIOC_EXT_PREPARE_BUF, v4l_ext_prepare_buf, v4l_print_ext_buffer, INFO_FL_QUEUE),
      IOCTL_INFO(VIDIOC_ENUM_DV_TIMINGS, v4l_stub_enum_dv_timings, v4l_print_enum_dv_timings, INFO_FL_CLEAR(v4l2_enum_dv_timings, pad)),
      IOCTL_INFO(VIDIOC_QUERY_DV_TIMINGS, v4l_stub_query_dv_timings, v4l_print_dv_timings, INFO_FL_ALWAYS_COPY),
      IOCTL_INFO(VIDIOC_DV_TIMINGS_CAP, v4l_stub_dv_timings_cap, v4l_print_dv_timings_cap, INFO_FL_CLEAR(v4l2_dv_timings_cap, pad)),
include/media/v4l2-ioctl.h +30 -0
index 39ac07fbc7b7..f7e375d38602 100644
@@ -168,16 +168,28 @@ struct v4l2_fh;
  *   :ref:`VIDIOC_REQBUFS <vidioc_reqbufs>` ioctl
  * @vidioc_querybuf: pointer to the function that implements
  *   :ref:`VIDIOC_QUERYBUF <vidioc_querybuf>` ioctl
+ * @vidioc_ext_querybuf: pointer to the function that implements
+ *   :ref:`VIDIOC_EXT_QUERYBUF <vidioc_ext_querybuf>` ioctl
  * @vidioc_qbuf: pointer to the function that implements
  *   :ref:`VIDIOC_QBUF <vidioc_qbuf>` ioctl
+ * @vidioc_ext_qbuf: pointer to the function that implements
+ *   :ref:`VIDIOC_EXT_QBUF <vidioc_ext_qbuf>` ioctl
  * @vidioc_expbuf: pointer to the function that implements
  *   :ref:`VIDIOC_EXPBUF <vidioc_expbuf>` ioctl
+ * @vidioc_ext_expbuf: pointer to the function that implements
+ *   :ref:`VIDIOC_EXT_EXPBUF <vidioc_ext_expbuf>` ioctl
  * @vidioc_dqbuf: pointer to the function that implements
  *   :ref:`VIDIOC_DQBUF <vidioc_qbuf>` ioctl
+ * @vidioc_ext_dqbuf: pointer to the function that implements
+ *   :ref:`VIDIOC_EXT_DQBUF <vidioc_ext_qbuf>` ioctl
  * @vidioc_create_bufs: pointer to the function that implements
  *   :ref:`VIDIOC_CREATE_BUFS <vidioc_create_bufs>` ioctl
+ * @vidioc_ext_create_bufs: pointer to the function that implements
+ *   :ref:`VIDIOC_EXT_CREATE_BUFS <vidioc_ext_create_bufs>` ioctl
  * @vidioc_prepare_buf: pointer to the function that implements
  *   :ref:`VIDIOC_PREPARE_BUF <vidioc_prepare_buf>` ioctl
+ * @vidioc_ext_prepare_buf: pointer to the function that implements
+ *   :ref:`VIDIOC_EXT_PREPARE_BUF <vidioc_ext_prepare_buf>` ioctl
  * @vidioc_overlay: pointer to the function that implements
  *   :ref:`VIDIOC_OVERLAY <vidioc_overlay>` ioctl
  * @vidioc_g_fbuf: pointer to the function that implements
@@ -438,17 +450,29 @@ struct v4l2_ioctl_ops {
                            struct v4l2_requestbuffers *b);
      int (*vidioc_querybuf)(struct file *file, void *fh,
                             struct v4l2_buffer *b);
+     int (*vidioc_ext_querybuf)(struct file *file, void *fh,
+                                struct v4l2_ext_buffer *b);
      int (*vidioc_qbuf)(struct file *file, void *fh,
                         struct v4l2_buffer *b);
+     int (*vidioc_ext_qbuf)(struct file *file, void *fh,
+                            struct v4l2_ext_buffer *b);
      int (*vidioc_expbuf)(struct file *file, void *fh,
                           struct v4l2_exportbuffer *e);
+     int (*vidioc_ext_expbuf)(struct file *file, void *fh,
+                              struct v4l2_ext_exportbuffer *e);
      int (*vidioc_dqbuf)(struct file *file, void *fh,
                          struct v4l2_buffer *b);
+     int (*vidioc_ext_dqbuf)(struct file *file, void *fh,
+                             struct v4l2_ext_buffer *b);
 
      int (*vidioc_create_bufs)(struct file *file, void *fh,
                                struct v4l2_create_buffers *b);
+     int (*vidioc_ext_create_bufs)(struct file *file, void *fh,
+                                   struct v4l2_ext_create_buffers *b);
      int (*vidioc_prepare_buf)(struct file *file, void *fh,
                                struct v4l2_buffer *b);
+     int (*vidioc_ext_prepare_buf)(struct file *file, void *fh,
+                                   struct v4l2_ext_buffer *b);
 
      int (*vidioc_overlay)(struct file *file, void *fh, unsigned int i);
      int (*vidioc_g_fbuf)(struct file *file, void *fh,
@@ -757,4 +781,10 @@ int v4l2_ext_format_to_format(const struct v4l2_ext_format *e,
                            struct v4l2_format *f,
                            bool mplane_cap, bool strict);
 
+int v4l2_ext_buffer_to_buffer(const struct v4l2_ext_buffer *e,
+                           struct v4l2_buffer *b,
+                           bool mplane_cap);
+int v4l2_buffer_to_ext_buffer(const struct v4l2_buffer *b,
+                           struct v4l2_ext_buffer *e);
+
 #endif /* _V4L2_IOCTL_H */
include/uapi/linux/videodev2.h +130 -1
index c7b169de1c7b..33c8348df13f 100644
@@ -953,6 +953,49 @@ struct v4l2_plane {
      __u32                   reserved[11];
 };
 
+/**
+ * struct v4l2_ext_plane - extended plane buffer info
+ * @bytesused: number of bytes occupied by data in the plane (payload)
+ * @length: size of this plane (NOT the payload) in bytes
+ * @mem_offset: when memory in the associated struct v4l2_ext_buffer is
+ *           V4L2_MEMORY_MMAP, equals the offset from the start of the
+ *           device memory for this plane (or is a "cookie" that should be
+ *           passed to mmap() called on the video node)
+ * @userptr: when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
+ *        to this plane
+ * @dmabuf.fd: when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
+ *          associated with this plane
+ * @dmabuf.offset: where the plane starts inside the DMABUF buffer. All planes
+ *              might share the same buffer object. In this case we need to
+ *              know where the plane start inside this buffer.
+ * @data_offset: offset in the plane to the start of data; usually 0, unless
+ *            there is a header in front of the data. data_offset is
+ *            relative to start_offset, so absolute data_offset is actually
+ *            start_offset + data_offset
+ *
+ *
+ * Multi-planar buffers consist of one or more planes, e.g. an YCbCr buffer
+ * with two planes can have one plane for Y, and another for interleaved CbCr
+ * components. Each plane can reside in a separate memory buffer, or even in
+ * a completely separate memory node (e.g. in embedded devices).
+ * Note that this struct is also used for uni-planar buffers, but in that case
+ * you'll only have one plane defined.
+ */
+struct v4l2_ext_plane {
+     __u32 bytesused;
+     __u32 length;
+     union {
+             __u32 mem_offset;
+             __u64 userptr;
+             struct {
+                     __s32 fd;
+                     __u32 offset;
+             } dmabuf;
+     } m;
+     __u32 data_offset;
+     __u32 reserved[10];
+};
+
 /**
  * struct v4l2_buffer - video buffer info
  * @index:   id number of the buffer
@@ -1010,6 +1053,40 @@ struct v4l2_buffer {
      };
 };
 
+/**
+ * struct v4l2_ext_buffer - extended video buffer info
+ * @index: id number of the buffer
+ * @type: enum v4l2_buf_type; buffer type. _MPLANE and _OVERLAY formats are
+ *     invalid
+ * @flags: buffer informational flags
+ * @field: enum v4l2_field; field order of the image in the buffer
+ * @timestamp: frame timestamp
+ * @sequence: sequence count of this frame
+ * @memory: enum v4l2_memory; the method, in which the actual video data is
+ *          passed
+ * @planes: per-plane buffer information
+ * @num_planes: number of plane buffers
+ * @request_fd: fd of the request that this buffer should use
+ * @reserved: some extra space reserved to add future fields (like timecode).
+ *         Must be set to 0
+ *
+ * Contains data exchanged by application and driver using one of the Streaming
+ * I/O methods.
+ */
+struct v4l2_ext_buffer {
+     __u32 index;
+     __u32 type;
+     __u32 flags;
+     __u32 field;
+     __u64 timestamp;
+     __u32 sequence;
+     __u32 memory;
+     struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
+     __u32 num_planes;
+     __u32 request_fd;
+     __u32 reserved[10];
+};
+
 /**
  * v4l2_timeval_to_ns - Convert timeval to nanoseconds
  * @ts:              pointer to the timeval variable to be converted
@@ -1087,6 +1164,35 @@ struct v4l2_exportbuffer {
      __u32           reserved[11];
 };
 
+/**
+ * struct v4l2_ext_exportbuffer - export of video buffer as DMABUF file
+ *                             descriptor using extended format
+ *
+ * @index: id number of the buffer
+ * @type: enum v4l2_buf_type; buffer type
+ * @flags: flags for newly created file(s), currently only O_CLOEXEC is
+ *      supported, refer to manual of open syscall for more details
+ * @first_plane: first plane to export. Most likely set to 0
+ * @num_planes: number of planes to export. Most set to the number of planes
+ *           attached to the buffer
+ * @fds: file descriptors associated with DMABUF (set by driver). Note that all
+ *    planes might share the same buffer and then be returned the same FD
+ *
+ * Contains data used for exporting a video buffer as DMABUF file descriptor.
+ * The buffer is identified by a 'cookie' returned by VIDIOC_QUERYBUF
+ * (identical to the cookie used to mmap() the buffer to userspace). All
+ * reserved fields must be set to zero.
+ */
+struct v4l2_ext_exportbuffer {
+     __u32 type; /* enum v4l2_buf_type */
+     __u32 index;
+     __u32 flags;
+     __u32 first_plane;
+     __u32 num_planes;
+     __s32 fds[VIDEO_MAX_PLANES];
+     __u32 reserved;
+};
+
 /*
  *   O V E R L A Y   P R E V I E W
  */
@@ -2483,6 +2589,23 @@ struct v4l2_create_buffers {
      __u32                   reserved[7];
 };
 
+/**
+ * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
+ * @index:   on return, index of the first created buffer
+ * @count:   entry: number of requested buffers,
+ *           return: number of created buffers
+ * @memory:  enum v4l2_memory; buffer memory type
+ * @capabilities: capabilities of this buffer type.
+ * @format:  frame format, for which buffers are requested
+ */
+struct v4l2_ext_create_buffers {
+     __u32                   index;
+     __u32                   count;
+     __u32                   memory;
+     __u32                   capabilities;
+     struct v4l2_ext_format  format;
+};
+
 /*
  *   I O C T L   C O D E S   F O R   V I D E O   D E V I C E S
  *
@@ -2586,6 +2709,13 @@ struct v4l2_create_buffers {
 #define VIDIOC_G_EXT_FMT     _IOWR('V', 104, struct v4l2_ext_format)
 #define VIDIOC_S_EXT_FMT     _IOWR('V', 105, struct v4l2_ext_format)
 #define VIDIOC_TRY_EXT_FMT   _IOWR('V', 106, struct v4l2_ext_format)
+#define VIDIOC_EXT_CREATE_BUFS       _IOWR('V', 107, struct v4l2_ext_create_buffers)
+#define VIDIOC_EXT_QUERYBUF  _IOWR('V', 108, struct v4l2_ext_buffer)
+#define VIDIOC_EXT_QBUF              _IOWR('V', 109, struct v4l2_ext_buffer)
+#define VIDIOC_EXT_DQBUF     _IOWR('V', 110, struct v4l2_ext_buffer)
+#define VIDIOC_EXT_PREPARE_BUF       _IOWR('V', 111, struct v4l2_ext_buffer)
+#define VIDIOC_EXT_EXPBUF    _IOWR('V', 112, struct v4l2_ext_exportbuffer)
+
 /* Reminder: when adding new ioctls please add support for them to
    drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
 
--

2.21.0

#13 lore raw

RFC PATCH v3 3/6 media: videobuf2: Expose helpers to implement the _ext_fmt and _ext_buf hooks

In reply to Boris Brezillon

The VB2 layer is used by a lot of drivers. Patch it to support the _EXT_FMT and _EXT_BUF ioctls in order to ease conversion of existing drivers to these new APIs.

Note that internally, the VB2 core is now only using ext structs and old APIs are supported through conversion wrappers.

---
Changes in v3:
- Rebased on top of media/master (post 5.4-rc1)

Changes in v2:
- New patch

---
 .../media/common/videobuf2/videobuf2-core.c   |   2 +
 .../media/common/videobuf2/videobuf2-v4l2.c   | 534 +++++++++++-------
 include/media/videobuf2-core.h                |   6 +-
 include/media/videobuf2-v4l2.h                |  26 +-
 4 files changed, 352 insertions(+), 216 deletions(-)
drivers/media/common/videobuf2/videobuf2-core.c +2 -0
index 4489744fbbd9..898d01b4dcb4 100644
@@ -1181,6 +1181,7 @@ static int __prepare_dmabuf(struct vb2_buffer *vb)
 		vb->planes[plane].length = 0;
 		vb->planes[plane].m.fd = 0;
 		vb->planes[plane].data_offset = 0;
+		vb->planes[plane].dbuf_offset = 0;
 
 		/* Acquire each plane's memory */
 		mem_priv = call_ptr_memop(vb, attach_dmabuf,
@@ -1224,6 +1225,7 @@ static int __prepare_dmabuf(struct vb2_buffer *vb)
 		vb->planes[plane].length = planes[plane].length;
 		vb->planes[plane].m.fd = planes[plane].m.fd;
 		vb->planes[plane].data_offset = planes[plane].data_offset;
+		vb->planes[plane].dbuf_offset = planes[plane].dbuf_offset;
 	}
 
 	if (reacquired) {
drivers/media/common/videobuf2/videobuf2-v4l2.c +321 -213

743 lines not on this page · raw message

include/media/videobuf2-core.h +5 -1
index 640aabe69450..6411fb2c89fe 100644
@@ -152,6 +152,8 @@ struct vb2_mem_ops {
  * @mem_priv:	private data with this plane.
  * @dbuf:	dma_buf - shared buffer object.
  * @dbuf_mapped:	flag to show whether dbuf is mapped or not
+ * @dbuf_offset: offset where the plane starts. Usually 0, unless the buffer
+ *		 is shared by all planes of a multi-planar format.
  * @bytesused:	number of bytes occupied by data in the plane (payload).
  * @length:	size of this plane (NOT the payload) in bytes.
  * @min_length:	minimum required size of this plane (NOT the payload) in bytes.
@@ -175,6 +177,7 @@ struct vb2_plane {
 	void			*mem_priv;
 	struct dma_buf		*dbuf;
 	unsigned int		dbuf_mapped;
+	unsigned int		dbuf_offset;
 	unsigned int		bytesused;
 	unsigned int		length;
 	unsigned int		min_length;
@@ -440,7 +443,8 @@ struct vb2_ops {
  *			struct vb2_buffer.
  *			For V4L2 this is a &struct vb2_v4l2_buffer.
  * @fill_user_buffer:	given a &vb2_buffer fill in the userspace structure.
- *			For V4L2 this is a &struct v4l2_buffer.
+ *			For V4L2 this is a &struct v4l2_buffer or
+ *			&struct v4l2_ext_buffer.
  * @fill_vb2_buffer:	given a userspace structure, fill in the &vb2_buffer.
  *			If the userspace structure is invalid, then this op
  *			will return an error.
include/media/videobuf2-v4l2.h +24 -2
index 8a10889dc2fd..99eff4639dc5 100644
@@ -36,7 +36,7 @@
  * @planes:	plane information (userptr/fd, length, bytesused, data_offset).
  *
  * Should contain enough information to be able to cover all the fields
- * of &struct v4l2_buffer at ``videodev2.h``.
+ * of &struct v4l2_buffer and &struct v4l2_ext_buffer at ``videodev2.h``.
  */
 struct vb2_v4l2_buffer {
 	struct vb2_buffer	vb2_buf;
@@ -72,6 +72,7 @@ int vb2_find_timestamp(const struct vb2_queue *q, u64 timestamp,
 		       unsigned int start_idx);
 
 int vb2_querybuf(struct vb2_queue *q, struct v4l2_buffer *b);
+int vb2_ext_querybuf(struct vb2_queue *q, struct v4l2_ext_buffer *b);
 
 /**
  * vb2_reqbufs() - Wrapper for vb2_core_reqbufs() that also verifies
@@ -92,6 +93,8 @@ int vb2_reqbufs(struct vb2_queue *q, struct v4l2_requestbuffers *req);
  *		&v4l2_ioctl_ops->vidioc_create_bufs handler in driver
  */
 int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create);
+int vb2_ext_create_bufs(struct vb2_queue *q,
+			struct v4l2_ext_create_buffers *create);
 
 /**
  * vb2_prepare_buf() - Pass ownership of a buffer from userspace to the kernel
@@ -117,6 +120,8 @@ int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create);
  */
 int vb2_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
 		    struct v4l2_buffer *b);
+int vb2_ext_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
+			struct v4l2_ext_buffer *b);
 
 /**
  * vb2_qbuf() - Queue a buffer from userspace
@@ -143,6 +148,8 @@ int vb2_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
  */
 int vb2_qbuf(struct vb2_queue *q, struct media_device *mdev,
 	     struct v4l2_buffer *b);
+int vb2_ext_qbuf(struct vb2_queue *q, struct media_device *mdev,
+		 struct v4l2_ext_buffer *b);
 
 /**
  * vb2_expbuf() - Export a buffer as a file descriptor
@@ -154,6 +161,7 @@ int vb2_qbuf(struct vb2_queue *q, struct media_device *mdev,
  * from &v4l2_ioctl_ops->vidioc_expbuf handler in driver.
  */
 int vb2_expbuf(struct vb2_queue *q, struct v4l2_exportbuffer *eb);
+int vb2_ext_expbuf(struct vb2_queue *q, struct v4l2_ext_exportbuffer *eb);
 
 /**
  * vb2_dqbuf() - Dequeue a buffer to the userspace
@@ -180,6 +188,8 @@ int vb2_expbuf(struct vb2_queue *q, struct v4l2_exportbuffer *eb);
  * from &v4l2_ioctl_ops->vidioc_dqbuf handler in driver.
  */
 int vb2_dqbuf(struct vb2_queue *q, struct v4l2_buffer *b, bool nonblocking);
+int vb2_ext_dqbuf(struct vb2_queue *q, struct v4l2_ext_buffer *b,
+		  bool nonblocking);
 
 /**
  * vb2_streamon - start streaming
@@ -276,15 +286,27 @@ int vb2_ioctl_reqbufs(struct file *file, void *priv,
 			  struct v4l2_requestbuffers *p);
 int vb2_ioctl_create_bufs(struct file *file, void *priv,
 			  struct v4l2_create_buffers *p);
+int vb2_ioctl_ext_create_bufs(struct file *file, void *priv,
+			      struct v4l2_ext_create_buffers *p);
 int vb2_ioctl_prepare_buf(struct file *file, void *priv,
 			  struct v4l2_buffer *p);
+int vb2_ioctl_ext_prepare_buf(struct file *file, void *priv,
+			      struct v4l2_ext_buffer *p);
 int vb2_ioctl_querybuf(struct file *file, void *priv, struct v4l2_buffer *p);
+int vb2_ioctl_ext_querybuf(struct file *file, void *priv,
+			   struct v4l2_ext_buffer *p);
 int vb2_ioctl_qbuf(struct file *file, void *priv, struct v4l2_buffer *p);
+int vb2_ioctl_ext_qbuf(struct file *file, void *priv,
+		       struct v4l2_ext_buffer *p);
 int vb2_ioctl_dqbuf(struct file *file, void *priv, struct v4l2_buffer *p);
+int vb2_ioctl_ext_dqbuf(struct file *file, void *priv,
+			struct v4l2_ext_buffer *p);
 int vb2_ioctl_streamon(struct file *file, void *priv, enum v4l2_buf_type i);
 int vb2_ioctl_streamoff(struct file *file, void *priv, enum v4l2_buf_type i);
 int vb2_ioctl_expbuf(struct file *file, void *priv,
-	struct v4l2_exportbuffer *p);
+		     struct v4l2_exportbuffer *p);
+int vb2_ioctl_ext_expbuf(struct file *file, void *priv,
+			 struct v4l2_ext_exportbuffer *p);
 
 /* struct v4l2_file_operations helpers */
 
Signature · 1 line
2.21.0
#14 lore raw

RFC PATCH v3 4/6 media: mediabus: Add an helper to convert a ext_pix format to an mbus_fmt

In reply to Boris Brezillon

Just a new version of v4l2_fill_mbus_format() to deal with the new v4l2_ext_pix_format struct.
This is needed to convert the VIMC driver to the EXT_FMT/EXT_BUF iocts.

---
Changes in v3:
- Rebased on top of media/master (post 5.4-rc1)

Changes in v2:
- New patch

---
 include/media/v4l2-mediabus.h | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
include/media/v4l2-mediabus.h +22 -0
index 45f88f0248c4..00de5ae05e84 100644
@@ -182,4 +182,26 @@ v4l2_fill_mbus_format_mplane(struct v4l2_mbus_framefmt *mbus_fmt,
 	mbus_fmt->xfer_func = pix_mp_fmt->xfer_func;
 }
 
+/**
+ * v4l2_fill_mbus_format_ext - Ancillary routine that fills a &struct
+ *	v4l2_mbus_framefmt from a &struct v4l2_ext_pix_format.
+ *
+ * @mbus_fmt:	pointer to &struct v4l2_mbus_framefmt to be filled
+ * @pix_fmt:	pointer to &struct v4l2_ext_pix_format to be used as model
+ * @code:	data format code (from &enum v4l2_mbus_pixelcode)
+ */
+static inline void
+v4l2_fill_mbus_format_ext(struct v4l2_mbus_framefmt *mbus_fmt,
+			  const struct v4l2_ext_pix_format *pix_fmt, u32 code)
+{
+	mbus_fmt->width = pix_fmt->width;
+	mbus_fmt->height = pix_fmt->height;
+	mbus_fmt->field = pix_fmt->field;
+	mbus_fmt->colorspace = pix_fmt->colorspace;
+	mbus_fmt->ycbcr_enc = pix_fmt->ycbcr_enc;
+	mbus_fmt->quantization = pix_fmt->quantization;
+	mbus_fmt->xfer_func = pix_fmt->xfer_func;
+	mbus_fmt->code = code;
+}
+
 #endif
Signature · 1 line
2.21.0
#15 lore raw

RFC PATCH v3 5/6 media: vivid: Convert the capture and output drivers to EXT_FMT/EXT_BUF

In reply to Boris Brezillon

This should simplify things a bit as we now have a single implementation instead of the MPLANE and !MPLANE versions.

---
Changes in v3:
- Rebased on top of media/master (post 5.4-rc1)

Changes in v2:
- New patch

---
 drivers/media/platform/vivid/vivid-core.c    |  30 ++-
 drivers/media/platform/vivid/vivid-vid-cap.c | 171 +++++-----------
 drivers/media/platform/vivid/vivid-vid-cap.h |  15 +-
 drivers/media/platform/vivid/vivid-vid-out.c | 195 +++++++------------
 drivers/media/platform/vivid/vivid-vid-out.h |  15 +-
 5 files changed, 142 insertions(+), 284 deletions(-)
drivers/media/platform/vivid/vivid-core.c +12 -18
index 53315c8dd2bb..502c1bf6f272 100644
@@ -501,20 +501,14 @@ static const struct v4l2_ioctl_ops vivid_ioctl_ops = {
 	.vidioc_querycap		= vidioc_querycap,
 
 	.vidioc_enum_fmt_vid_cap	= vivid_enum_fmt_vid,
-	.vidioc_g_fmt_vid_cap		= vidioc_g_fmt_vid_cap,
-	.vidioc_try_fmt_vid_cap		= vidioc_try_fmt_vid_cap,
-	.vidioc_s_fmt_vid_cap		= vidioc_s_fmt_vid_cap,
-	.vidioc_g_fmt_vid_cap_mplane	= vidioc_g_fmt_vid_cap_mplane,
-	.vidioc_try_fmt_vid_cap_mplane	= vidioc_try_fmt_vid_cap_mplane,
-	.vidioc_s_fmt_vid_cap_mplane	= vidioc_s_fmt_vid_cap_mplane,
+	.vidioc_g_ext_fmt_vid_cap	= vivid_g_fmt_vid_cap,
+	.vidioc_try_ext_fmt_vid_cap	= vivid_try_fmt_vid_cap,
+	.vidioc_s_ext_fmt_vid_cap	= vivid_s_fmt_vid_cap,
 
 	.vidioc_enum_fmt_vid_out	= vivid_enum_fmt_vid,
-	.vidioc_g_fmt_vid_out		= vidioc_g_fmt_vid_out,
-	.vidioc_try_fmt_vid_out		= vidioc_try_fmt_vid_out,
-	.vidioc_s_fmt_vid_out		= vidioc_s_fmt_vid_out,
-	.vidioc_g_fmt_vid_out_mplane	= vidioc_g_fmt_vid_out_mplane,
-	.vidioc_try_fmt_vid_out_mplane	= vidioc_try_fmt_vid_out_mplane,
-	.vidioc_s_fmt_vid_out_mplane	= vidioc_s_fmt_vid_out_mplane,
+	.vidioc_g_ext_fmt_vid_out	= vivid_g_fmt_vid_out,
+	.vidioc_try_ext_fmt_vid_out	= vivid_try_fmt_vid_out,
+	.vidioc_s_ext_fmt_vid_out	= vivid_s_fmt_vid_out,
 
 	.vidioc_g_selection		= vidioc_g_selection,
 	.vidioc_s_selection		= vidioc_s_selection,
@@ -559,12 +553,12 @@ static const struct v4l2_ioctl_ops vivid_ioctl_ops = {
 	.vidioc_s_fbuf			= vidioc_s_fbuf,
 
 	.vidioc_reqbufs			= vb2_ioctl_reqbufs,
-	.vidioc_create_bufs		= vb2_ioctl_create_bufs,
-	.vidioc_prepare_buf		= vb2_ioctl_prepare_buf,
-	.vidioc_querybuf		= vb2_ioctl_querybuf,
-	.vidioc_qbuf			= vb2_ioctl_qbuf,
-	.vidioc_dqbuf			= vb2_ioctl_dqbuf,
-	.vidioc_expbuf			= vb2_ioctl_expbuf,
+	.vidioc_ext_create_bufs		= vb2_ioctl_ext_create_bufs,
+	.vidioc_ext_prepare_buf		= vb2_ioctl_ext_prepare_buf,
+	.vidioc_ext_querybuf		= vb2_ioctl_ext_querybuf,
+	.vidioc_ext_qbuf		= vb2_ioctl_ext_qbuf,
+	.vidioc_ext_dqbuf		= vb2_ioctl_ext_dqbuf,
+	.vidioc_ext_expbuf		= vb2_ioctl_ext_expbuf,
 	.vidioc_streamon		= vb2_ioctl_streamon,
 	.vidioc_streamoff		= vb2_ioctl_streamoff,
 
drivers/media/platform/vivid/vivid-vid-cap.c +53 -118

292 lines not on this page · raw message

drivers/media/platform/vivid/vivid-vid-cap.h +6 -9
index 1e422a59eeab..7c9fc5c787b5 100644
@@ -17,15 +17,12 @@ extern const char * const vivid_ctrl_standard_strings[];
 
 extern const struct vb2_ops vivid_vid_cap_qops;
 
-int vivid_g_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f);
-int vivid_try_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f);
-int vivid_s_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_g_fmt_vid_cap_mplane(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_try_fmt_vid_cap_mplane(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_s_fmt_vid_cap_mplane(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_g_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_try_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_s_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f);
+int vivid_g_fmt_vid_cap(struct file *file, void *priv,
+			struct v4l2_ext_pix_format *f);
+int vivid_try_fmt_vid_cap(struct file *file, void *priv,
+			  struct v4l2_ext_pix_format *f);
+int vivid_s_fmt_vid_cap(struct file *file, void *priv,
+			struct v4l2_ext_pix_format *f);
 int vivid_vid_cap_g_selection(struct file *file, void *priv, struct v4l2_selection *sel);
 int vivid_vid_cap_s_selection(struct file *file, void *fh, struct v4l2_selection *s);
 int vivid_vid_cap_g_pixelaspect(struct file *file, void *priv, int type, struct v4l2_fract *f);
drivers/media/platform/vivid/vivid-vid-out.c +65 -130

313 lines not on this page · raw message

drivers/media/platform/vivid/vivid-vid-out.h +6 -9
index 8d56314f4ea1..b84dc578af36 100644
@@ -12,15 +12,12 @@ extern const struct vb2_ops vivid_vid_out_qops;
 
 void vivid_update_format_out(struct vivid_dev *dev);
 
-int vivid_g_fmt_vid_out(struct file *file, void *priv, struct v4l2_format *f);
-int vivid_try_fmt_vid_out(struct file *file, void *priv, struct v4l2_format *f);
-int vivid_s_fmt_vid_out(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_g_fmt_vid_out_mplane(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_try_fmt_vid_out_mplane(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_s_fmt_vid_out_mplane(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_g_fmt_vid_out(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_try_fmt_vid_out(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_s_fmt_vid_out(struct file *file, void *priv, struct v4l2_format *f);
+int vivid_g_fmt_vid_out(struct file *file, void *priv,
+			struct v4l2_ext_pix_format *f);
+int vivid_try_fmt_vid_out(struct file *file, void *priv,
+			  struct v4l2_ext_pix_format *f);
+int vivid_s_fmt_vid_out(struct file *file, void *priv,
+			struct v4l2_ext_pix_format *f);
 int vivid_vid_out_g_selection(struct file *file, void *priv, struct v4l2_selection *sel);
 int vivid_vid_out_s_selection(struct file *file, void *fh, struct v4l2_selection *s);
 int vivid_vid_out_g_pixelaspect(struct file *file, void *priv, int type, struct v4l2_fract *f);
Signature · 1 line
2.21.0
#16 lore raw

RFC PATCH v3 6/6 media: vimc: Implement the ext_fmt and ext_buf hooks

In reply to Boris Brezillon

Convert the driver to the _ext_fmt and _ext_buf API.

---
Changes in v3:
- Rebased on top of media/master (post 5.4-rc1)

Changes in v2:
- New patch

---
 drivers/media/platform/vimc/vimc-capture.c | 65 +++++++++++-----------
 drivers/media/platform/vimc/vimc-common.c  |  4 +-
 drivers/media/platform/vimc/vimc-common.h  |  2 +-
 3 files changed, 36 insertions(+), 35 deletions(-)
drivers/media/platform/vimc/vimc-capture.c +33 -32
index 602f80323031..8a9d0039bf6e 100644
@@ -16,7 +16,7 @@ struct vimc_cap_device {
 	struct vimc_ent_device ved;
 	struct video_device vdev;
 	struct device *dev;
-	struct v4l2_pix_format format;
+	struct v4l2_ext_pix_format format;
 	struct vb2_queue queue;
 	struct list_head buf_list;
 	/*
@@ -32,12 +32,13 @@ struct vimc_cap_device {
 	struct vimc_stream stream;
 };
 
-static const struct v4l2_pix_format fmt_default = {
+static const struct v4l2_ext_pix_format fmt_default = {
 	.width = 640,
 	.height = 480,
 	.pixelformat = V4L2_PIX_FMT_RGB24,
 	.field = V4L2_FIELD_NONE,
 	.colorspace = V4L2_COLORSPACE_DEFAULT,
+	.num_planes = 1,
 };
 
 struct vimc_cap_buffer {
@@ -63,7 +64,7 @@ static int vimc_cap_querycap(struct file *file, void *priv,
 }
 
 static void vimc_cap_get_format(struct vimc_ent_device *ved,
-				struct v4l2_pix_format *fmt)
+				struct v4l2_ext_pix_format *fmt)
 {
 	struct vimc_cap_device *vcap = container_of(ved, struct vimc_cap_device,
 						    ved);
@@ -72,19 +73,18 @@ static void vimc_cap_get_format(struct vimc_ent_device *ved,
 }
 
 static int vimc_cap_g_fmt_vid_cap(struct file *file, void *priv,
-				  struct v4l2_format *f)
+				  struct v4l2_ext_pix_format *f)
 {
 	struct vimc_cap_device *vcap = video_drvdata(file);
 
-	f->fmt.pix = vcap->format;
+	*f = vcap->format;
 
 	return 0;
 }
 
 static int vimc_cap_try_fmt_vid_cap(struct file *file, void *priv,
-				    struct v4l2_format *f)
+				    struct v4l2_ext_pix_format *format)
 {
-	struct v4l2_pix_format *format = &f->fmt.pix;
 	const struct vimc_pix_map *vpix;
 
 	format->width = clamp_t(u32, format->width, VIMC_FRAME_MIN_WIDTH,
@@ -99,8 +99,10 @@ static int vimc_cap_try_fmt_vid_cap(struct file *file, void *priv,
 		vpix = vimc_pix_map_by_pixelformat(format->pixelformat);
 	}
 	/* TODO: Add support for custom bytesperline values */
-	format->bytesperline = format->width * vpix->bpp;
-	format->sizeimage = format->bytesperline * format->height;
+	format->num_planes = 1;
+	format->plane_fmt[0].bytesperline = format->width * vpix->bpp;
+	format->plane_fmt[0].sizeimage = format->plane_fmt[0].bytesperline *
+					 format->height;
 
 	if (format->field == V4L2_FIELD_ANY)
 		format->field = fmt_default.field;
@@ -111,7 +113,7 @@ static int vimc_cap_try_fmt_vid_cap(struct file *file, void *priv,
 }
 
 static int vimc_cap_s_fmt_vid_cap(struct file *file, void *priv,
-				  struct v4l2_format *f)
+				  struct v4l2_ext_pix_format *f)
 {
 	struct vimc_cap_device *vcap = video_drvdata(file);
 	int ret;
@@ -133,12 +135,10 @@ static int vimc_cap_s_fmt_vid_cap(struct file *file, void *priv,
 		vcap->format.quantization, vcap->format.xfer_func,
 		vcap->format.ycbcr_enc,
 		/* new */
-		f->fmt.pix.width, f->fmt.pix.height,
-		f->fmt.pix.pixelformat,	f->fmt.pix.colorspace,
-		f->fmt.pix.quantization, f->fmt.pix.xfer_func,
-		f->fmt.pix.ycbcr_enc);
+		f->width, f->height, f->pixelformat, f->colorspace,
+		f->quantization, f->xfer_func, f->ycbcr_enc);
 
-	vcap->format = f->fmt.pix;
+	vcap->format = *f;
 
 	return 0;
 }
@@ -193,19 +193,19 @@ static const struct v4l2_file_operations vimc_cap_fops = {
 static const struct v4l2_ioctl_ops vimc_cap_ioctl_ops = {
 	.vidioc_querycap = vimc_cap_querycap,
 
-	.vidioc_g_fmt_vid_cap = vimc_cap_g_fmt_vid_cap,
-	.vidioc_s_fmt_vid_cap = vimc_cap_s_fmt_vid_cap,
-	.vidioc_try_fmt_vid_cap = vimc_cap_try_fmt_vid_cap,
+	.vidioc_g_ext_fmt_vid_cap = vimc_cap_g_fmt_vid_cap,
+	.vidioc_s_ext_fmt_vid_cap = vimc_cap_s_fmt_vid_cap,
+	.vidioc_try_ext_fmt_vid_cap = vimc_cap_try_fmt_vid_cap,
 	.vidioc_enum_fmt_vid_cap = vimc_cap_enum_fmt_vid_cap,
 	.vidioc_enum_framesizes = vimc_cap_enum_framesizes,
 
 	.vidioc_reqbufs = vb2_ioctl_reqbufs,
-	.vidioc_create_bufs = vb2_ioctl_create_bufs,
-	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
-	.vidioc_querybuf = vb2_ioctl_querybuf,
-	.vidioc_qbuf = vb2_ioctl_qbuf,
-	.vidioc_dqbuf = vb2_ioctl_dqbuf,
-	.vidioc_expbuf = vb2_ioctl_expbuf,
+	.vidioc_ext_create_bufs = vb2_ioctl_ext_create_bufs,
+	.vidioc_ext_prepare_buf = vb2_ioctl_ext_prepare_buf,
+	.vidioc_ext_querybuf = vb2_ioctl_ext_querybuf,
+	.vidioc_ext_qbuf = vb2_ioctl_ext_qbuf,
+	.vidioc_ext_dqbuf = vb2_ioctl_ext_dqbuf,
+	.vidioc_ext_expbuf = vb2_ioctl_ext_expbuf,
 	.vidioc_streamon = vb2_ioctl_streamon,
 	.vidioc_streamoff = vb2_ioctl_streamoff,
 };
@@ -286,10 +286,11 @@ static int vimc_cap_queue_setup(struct vb2_queue *vq, unsigned int *nbuffers,
 	struct vimc_cap_device *vcap = vb2_get_drv_priv(vq);
 
 	if (*nplanes)
-		return sizes[0] < vcap->format.sizeimage ? -EINVAL : 0;
+		return sizes[0] < vcap->format.plane_fmt[0].sizeimage ?
+		       -EINVAL : 0;
 	/* We don't support multiplanes for now */
 	*nplanes = 1;
-	sizes[0] = vcap->format.sizeimage;
+	sizes[0] = vcap->format.plane_fmt[0].sizeimage;
 
 	return 0;
 }
@@ -297,7 +298,7 @@ static int vimc_cap_queue_setup(struct vb2_queue *vq, unsigned int *nbuffers,
 static int vimc_cap_buffer_prepare(struct vb2_buffer *vb)
 {
 	struct vimc_cap_device *vcap = vb2_get_drv_priv(vb->vb2_queue);
-	unsigned long size = vcap->format.sizeimage;
+	unsigned long size = vcap->format.plane_fmt[0].sizeimage;
 
 	if (vb2_plane_size(vb, 0) < size) {
 		dev_err(vcap->dev, "%s: buffer too small (%lu < %lu)\n",
@@ -374,11 +375,11 @@ static void *vimc_cap_process_frame(struct vimc_ent_device *ved,
 
 	vbuf = vb2_plane_vaddr(&vimc_buf->vb2.vb2_buf, 0);
 
-	memcpy(vbuf, frame, vcap->format.sizeimage);
+	memcpy(vbuf, frame, vcap->format.plane_fmt[0].sizeimage);
 
 	/* Set it as ready */
 	vb2_set_plane_payload(&vimc_buf->vb2.vb2_buf, 0,
-			      vcap->format.sizeimage);
+			      vcap->format.plane_fmt[0].sizeimage);
 	vb2_buffer_done(&vimc_buf->vb2.vb2_buf, VB2_BUF_STATE_DONE);
 	return NULL;
 }
@@ -443,9 +444,9 @@ struct vimc_ent_device *vimc_cap_add(struct vimc_device *vimc,
 	/* Set default frame format */
 	vcap->format = fmt_default;
 	vpix = vimc_pix_map_by_pixelformat(vcap->format.pixelformat);
-	vcap->format.bytesperline = vcap->format.width * vpix->bpp;
-	vcap->format.sizeimage = vcap->format.bytesperline *
-				 vcap->format.height;
+	vcap->format.plane_fmt[0].bytesperline = vcap->format.width * vpix->bpp;
+	vcap->format.plane_fmt[0].sizeimage = vcap->format.plane_fmt[0].bytesperline *
+					      vcap->format.height;
 
 	/* Fill the vimc_ent_device struct */
 	vcap->ved.ent = &vcap->vdev.entity;
drivers/media/platform/vimc/vimc-common.c +2 -2
index a3120f4f7a90..c10cb3e9b934 100644
@@ -268,14 +268,14 @@ static int vimc_get_mbus_format(struct media_pad *pad,
 							 entity);
 		struct vimc_ent_device *ved = video_get_drvdata(vdev);
 		const struct vimc_pix_map *vpix;
-		struct v4l2_pix_format vdev_fmt;
+		struct v4l2_ext_pix_format vdev_fmt;
 
 		if (!ved->vdev_get_format)
 			return -ENOIOCTLCMD;
 
 		ved->vdev_get_format(ved, &vdev_fmt);
 		vpix = vimc_pix_map_by_pixelformat(vdev_fmt.pixelformat);
-		v4l2_fill_mbus_format(&fmt->format, &vdev_fmt, vpix->code);
+		v4l2_fill_mbus_format_ext(&fmt->format, &vdev_fmt, vpix->code);
 	} else {
 		return -EINVAL;
 	}
drivers/media/platform/vimc/vimc-common.h +1 -1
index 698db7c07645..21bbe31c8029 100644
@@ -113,7 +113,7 @@ struct vimc_ent_device {
 	void * (*process_frame)(struct vimc_ent_device *ved,
 				const void *frame);
 	void (*vdev_get_format)(struct vimc_ent_device *ved,
-			      struct v4l2_pix_format *fmt);
+				struct v4l2_ext_pix_format *fmt);
 };
 
 /**
Signature · 1 line
2.21.0
#17 lore raw
In reply to Boris Brezillon

On Tue, 8 Oct 2019 11:11:13 +0200
Boris Brezillon <[email protected]> wrote:

Oops, forgot to update the subject. Should be:

"[RFC PATCH v3 0/6] media: v4l2: Add extended fmt and buffer ioctls"

Quoted 98 lines

Hello,

This RFC follows the discussion started by Hans [1] a few months back. It does not try to address all the problem reported in this thread but instead focuses on the FMT and BUF(S) ioctls.

Note that my primary goal is to unify handling for multiplanar and singleplanar formats and extend things to support the "single dmabuf storing all pixel planes" issue.

This version received a bit more testing than the previous one (added new tests to v4l2-compliance [2] to make sure EXT ioctls work as expected and also checked that !ext -> ext wrappers work correctly by running the old tests). Note that I'm not planning to post those v4l-utils patches on the ML until we've settled down on the userspace API, unless I'm explicitly asked to do so.

Right now I'm focusing on the case I was primarily interested in:
single dmabuf storing all pixel planes (each being at a different offset), and it seems that patching the VB2 core to support that is not a trivial task.

So here are a few questions for V4L/DMABUF experts:
- Can the same dmabuf be mapped several times. I think it's okay apart from the extra/needless time spent doing cache maintenance operations, but there might be issues if an IOMMU is involved (duplicate mappings?). If it's not okay, then we need to find a solution to only attach/map the DMABUF once when it's used for several planes (this is what I tried to do here [3], but I'm not entirely happy with the implementation and started to investigate another approach here [4]).
- How should we pass the offset to drivers that were previously using the get_cookie() (or the dma_sg wrapper) to retrieve an sg table. Adding the offset to the dma_addr or vaddr for vmalloc or dma-contig case can be done in the core, but for an sg-table it's a bit more complicated. Should drivers access this piece of information directly from vb2_plane->dbuf_offset? And in that case, how do we make sure drivers don't simply ignore the offset and assume it's always zero?

Few words about the feedback I got from Brian and Nicolas on my v1:

- modifier field has been moved to v4l2_ext_format as suggested
- v4l2_timecode is still not present in v4l2_ext_buffer, but can be added back thanks to the extra reserved space
- the ENUMFMT is left as is for now, because I think we want Maxime's work on DRM/V4L format unification to land before reworking the ioctl (exposing extra info about the format and not only the 4CC code?). That also means that there's currently no way to know which modifiers are supported
- EXT_FMT/EXT_BUF capability flags to detect whether new ioctls are supported or not have not been added yet

Nothing has changed in v3, just rebased patches on top of media/master so we can discuss it during the Media Summit.

Regards,

Boris

[1]https://www.mail-archive.com/[email protected]/msg135729.html [2]https://github.com/bbrezillon/v4l-utils/commits/master [3]https://github.com/bbrezillon/linux/commit/4882435f80b05a61827649d55cc0f0cee79680a7 [4]https://github.com/bbrezillon/linux/commit/a415216c6aaab2d51f0bd62270b994c8196ddd90

Boris Brezillon (5):
media: v4l2: Extend pixel formats to unify single/multi-planar
handling (and more)
media: videobuf2: Expose helpers to implement the _ext_fmt and
_ext_buf hooks
media: mediabus: Add an helper to convert a ext_pix format to an
mbus_fmt
media: vivid: Convert the capture and output drivers to
EXT_FMT/EXT_BUF
media: vimc: Implement the ext_fmt and ext_buf hooks

Hans Verkuil (1):
media: v4l2: Add extended buffer operations

 .../media/common/videobuf2/videobuf2-core.c   |    2 +
 .../media/common/videobuf2/videobuf2-v4l2.c   |  534 ++++----
 drivers/media/platform/vimc/vimc-capture.c    |   65 +-
 drivers/media/platform/vimc/vimc-common.c     |    4 +-
 drivers/media/platform/vimc/vimc-common.h     |    2 +-
 drivers/media/platform/vivid/vivid-core.c     |   30 +-
 drivers/media/platform/vivid/vivid-vid-cap.c  |  171 +--
 drivers/media/platform/vivid/vivid-vid-cap.h  |   15 +-
 drivers/media/platform/vivid/vivid-vid-out.c  |  195 +--
 drivers/media/platform/vivid/vivid-vid-out.h  |   15 +-
 drivers/media/v4l2-core/v4l2-dev.c            |   54 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          | 1127 +++++++++++++++--
 include/media/v4l2-ioctl.h                    |   63 +
 include/media/v4l2-mediabus.h                 |   22 +
 include/media/videobuf2-core.h                |    6 +-
 include/media/videobuf2-v4l2.h                |   26 +-
 include/uapi/linux/videodev2.h                |  211 +++
 17 files changed, 1897 insertions(+), 645 deletions(-)
#18 lore raw
In reply to Boris Brezillon

Hi Boris,

Quoted 10 lines · Boris Brezillon

Hello,

This RFC follows the discussion started by Hans [1] a few months back. It does not try to address all the problem reported in this thread but instead focuses on the FMT and BUF(S) ioctls.

Note that my primary goal is to unify handling for multiplanar and singleplanar formats and extend things to support the "single dmabuf storing all pixel planes" issue.

Quoted 7 lines

This version received a bit more testing than the previous one (added new tests to v4l2-compliance [2] to make sure EXT ioctls work as expected and also checked that !ext -> ext wrappers work correctly by running the old tests). Note that I'm not planning to post those v4l-utils patches on the ML until we've settled down on the userspace API, unless I'm explicitly asked to do so.

I'm new to V4L2, so I'm sorry if some of my questions don't make sense.

The venus codec can decode to UBWC format so I'm working with this patch set to see how the modifiers flow will go. I've been able to integrate these patches for the most part.

Is the expectation with these new ioctls that they will work seamlessly with all codecs? Or that a codec can implement only the ones that are needed? i.e. I only need to pass the modifier, so could I only implement S_EXT_FMT? And then is the expectation that I know that for EXT I set the queue to !MPLANE?

I find the interaction between _MPLANE and !MPLANE to still be a odd with the new ioctls. I had thought that I could use the same queue designation for all the calls. But for the codec I'm looking at (venus) it is MPLANE only, so I call REQBUFS with V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE and all of the new ioctls with V4L2_BUF_TYPE_VIDEO_CAPTURE. I'm failing to see how to do this in a backward compatible way.

Quoted 83 lines

Right now I'm focusing on the case I was primarily interested in:
single dmabuf storing all pixel planes (each being at a different offset), and it seems that patching the VB2 core to support that is not a trivial task.

So here are a few questions for V4L/DMABUF experts:
- Can the same dmabuf be mapped several times. I think it's okay apart from the extra/needless time spent doing cache maintenance operations, but there might be issues if an IOMMU is involved (duplicate mappings?). If it's not okay, then we need to find a solution to only attach/map the DMABUF once when it's used for several planes (this is what I tried to do here [3], but I'm not entirely happy with the implementation and started to investigate another approach here [4]).
- How should we pass the offset to drivers that were previously using the get_cookie() (or the dma_sg wrapper) to retrieve an sg table. Adding the offset to the dma_addr or vaddr for vmalloc or dma-contig case can be done in the core, but for an sg-table it's a bit more complicated. Should drivers access this piece of information directly from vb2_plane->dbuf_offset? And in that case, how do we make sure drivers don't simply ignore the offset and assume it's always zero?

Few words about the feedback I got from Brian and Nicolas on my v1:

- modifier field has been moved to v4l2_ext_format as suggested
- v4l2_timecode is still not present in v4l2_ext_buffer, but can be added back thanks to the extra reserved space
- the ENUMFMT is left as is for now, because I think we want Maxime's work on DRM/V4L format unification to land before reworking the ioctl (exposing extra info about the format and not only the 4CC code?). That also means that there's currently no way to know which modifiers are supported
- EXT_FMT/EXT_BUF capability flags to detect whether new ioctls are supported or not have not been added yet

Nothing has changed in v3, just rebased patches on top of media/master so we can discuss it during the Media Summit.

Regards,

Boris

[1]https://www.mail-archive.com/[email protected]/msg135729.html [2]https://github.com/bbrezillon/v4l-utils/commits/master [3]https://github.com/bbrezillon/linux/commit/4882435f80b05a61827649d55cc0f0cee79680a7 [4]https://github.com/bbrezillon/linux/commit/a415216c6aaab2d51f0bd62270b994c8196ddd90

Boris Brezillon (5):
media: v4l2: Extend pixel formats to unify single/multi-planar
handling (and more)
media: videobuf2: Expose helpers to implement the _ext_fmt and
_ext_buf hooks
media: mediabus: Add an helper to convert a ext_pix format to an
mbus_fmt
media: vivid: Convert the capture and output drivers to
EXT_FMT/EXT_BUF
media: vimc: Implement the ext_fmt and ext_buf hooks

Hans Verkuil (1):
media: v4l2: Add extended buffer operations

 .../media/common/videobuf2/videobuf2-core.c   |    2 +
 .../media/common/videobuf2/videobuf2-v4l2.c   |  534 ++++----
 drivers/media/platform/vimc/vimc-capture.c    |   65 +-
 drivers/media/platform/vimc/vimc-common.c     |    4 +-
 drivers/media/platform/vimc/vimc-common.h     |    2 +-
 drivers/media/platform/vivid/vivid-core.c     |   30 +-
 drivers/media/platform/vivid/vivid-vid-cap.c  |  171 +--
 drivers/media/platform/vivid/vivid-vid-cap.h  |   15 +-
 drivers/media/platform/vivid/vivid-vid-out.c  |  195 +--
 drivers/media/platform/vivid/vivid-vid-out.h  |   15 +-
 drivers/media/v4l2-core/v4l2-dev.c            |   54 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          | 1127 +++++++++++++++--
 include/media/v4l2-ioctl.h                    |   63 +
 include/media/v4l2-mediabus.h                 |   22 +
 include/media/videobuf2-core.h                |    6 +-
 include/media/videobuf2-v4l2.h                |   26 +-
 include/uapi/linux/videodev2.h                |  211 +++
 17 files changed, 1897 insertions(+), 645 deletions(-)
Signature · 1 line
2.21.0