media: videobuf2: Allow applications customize data offsets of capture buffers
Subsystems (2)
PATCH 1/1 media: videobuf2: Allow applications customize data offsets of capture buffers
Use data offsets provided by applications for multi-planar capture buffers. This allows V4L to import and use dma-bufs exported by other subsystems in cases where application wants to customize data offsets of capture buffers in order to meet hardware alignment requirements of both dma-buf exporter and importer.
This feature is wanted for providing a better support of media hardware found on Chromebooks. In particular display and camera ISP hardware of Rockchip and MediaTek SoCs require special handling by userspace because display h/w has specific alignment requirements that don't match default alignments expected by V4L and there is a need to customize the data offsets in case of multi-planar formats.
Some drivers already have preliminary support for data offsets customization of capture buffers, like NVIDIA Tegra video decoder driver for example, and V4L allows applications to provide data offsets for multi-planar output buffers, let's support such customization for the capture buffers as well.
--- Documentation/userspace-api/media/v4l/buffer.rst | 9 ++++++++- drivers/media/common/videobuf2/videobuf2-v4l2.c | 7 +++++++ 2 files changed, 15 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/v4l/buffer.rst +8 -1
@@ -369,13 +369,20 @@ struct v4l2_plane - ``data_offset`` - Offset in bytes to video data in the plane. Drivers must set this field when ``type`` refers to a capture stream, applications when - it refers to an output stream. + it refers to an output or capture stream. .. note:: That data_offset is included in ``bytesused``. So the size of the image in the plane is ``bytesused``-``data_offset`` at offset ``data_offset`` from the start of the plane. + + For capture planes data_offset may be specified by applications + and by drivers. Driver may override application's offset or error + out buffer if offset can't be satisfied by hardware. This allows + applications to customize data offsets of imported dma-bufs. + Handling of application's offsets is driver-dependent, application + must use the resulting buffer offset. * - __u32 - ``reserved[11]`` - Reserved for future use. Should be zeroed by drivers and
drivers/media/common/videobuf2/videobuf2-v4l2.c +7 -0
@@ -263,6 +263,13 @@ static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb, struct v4l2_buffer *b psrc->bytesused : pdst->length; pdst->data_offset = psrc->data_offset; } + }else { +for (plane =0 ; plane < vb->num_planes; ++plane) { +struct vb2_plane *pdst = &planes[plane]; +struct v4l2_plane *psrc = &b->m.planes[plane]; + + pdst->data_offset = psrc->data_offset; + } } }else {/*
Signature · 1 line
2.35.1
Hi Dmitry,
thanks for giving a second look a this issue.
Le mardi 22 mars 2022 à 16:23 +0300, Dmitry Osipenko a écrit :
Quoted 30 lines
Use data offsets provided by applications for multi-planar capture buffers. This allows V4L to import and use dma-bufs exported by other subsystems in cases where application wants to customize data offsets of capture buffers in order to meet hardware alignment requirements of both dma-buf exporter and importer.
This feature is wanted for providing a better support of media hardware found on Chromebooks. In particular display and camera ISP hardware of Rockchip and MediaTek SoCs require special handling by userspace because display h/w has specific alignment requirements that don't match default alignments expected by V4L and there is a need to customize the data offsets in case of multi-planar formats.
Some drivers already have preliminary support for data offsets customization of capture buffers, like NVIDIA Tegra video decoder driver for example, and V4L allows applications to provide data offsets for multi-planar output buffers, let's support such customization for the capture buffers as well.
Signed-off-by Dmitry Osipenko <[email protected]>--- Documentation/userspace-api/media/v4l/buffer.rst | 9 ++++++++- drivers/media/common/videobuf2/videobuf2-v4l2.c | 7 +++++++ 2 files changed, 15 insertions(+), 1 deletion(-)Documentation/userspace-api/media/v4l/buffer.rst +0
-0
Documentation/userspace-api/media/v4l/buffer.rst struct v4l2_plane · line 372
Quoted 5 lines
Documentation/userspace-api/media/v4l/buffer.rst +1
-1@@ -369,13 +369,20 @@ struct v4l2_plane - ``data_offset`` - Offset in bytes to video data in the plane. Drivers must set this field when ``type`` refers to a capture stream, applications when - it refers to an output stream. + it refers to an output or capture stream.
There is a clear contradiction in this paragraph. Both the driver and the application MUST set the data_offset.
Would it be possible to demo your idea by implementing this in a virtual driver ? vivid already have data_offset for capture in some cases, you could verify if your idea works without any conflict in this scenario.
Quoted 34 lines
Documentation/userspace-api/media/v4l/buffer.rst +7
-0.. note:: That data_offset is included in ``bytesused``. So the size of the image in the plane is ``bytesused``-``data_offset`` at offset ``data_offset`` from the start of the plane. + + For capture planes data_offset may be specified by applications + and by drivers. Driver may override application's offset or error + out buffer if offset can't be satisfied by hardware. This allows + applications to customize data offsets of imported dma-bufs. + Handling of application's offsets is driver-dependent, application + must use the resulting buffer offset. * - __u32 - ``reserved[11]`` - Reserved for future use. Should be zeroed by drivers anddrivers/media/common/videobuf2/videobuf2-v4l2.c +7
-0@@ -263,6 +263,13 @@ static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb, struct v4l2_buffer *b psrc->bytesused : pdst->length; pdst->data_offset = psrc->data_offset; } + }else { +for (plane =0 ; plane < vb->num_planes; ++plane) { +struct vb2_plane *pdst = &planes[plane]; +struct v4l2_plane *psrc = &b->m.planes[plane]; + + pdst->data_offset = psrc->data_offset; + } } }else {/*
Hi Nicolas,
Quoted 30 lines · Nicolas Dufresne
Hi Dmitry,
thanks for giving a second look a this issue.
Le mardi 22 mars 2022 à 16:23 +0300, Dmitry Osipenko a écrit :
Quoted 25 lines
Use data offsets provided by applications for multi-planar capture buffers. This allows V4L to import and use dma-bufs exported by other subsystems in cases where application wants to customize data offsets of capture buffers in order to meet hardware alignment requirements of both dma-buf exporter and importer.
This feature is wanted for providing a better support of media hardware found on Chromebooks. In particular display and camera ISP hardware of Rockchip and MediaTek SoCs require special handling by userspace because display h/w has specific alignment requirements that don't match default alignments expected by V4L and there is a need to customize the data offsets in case of multi-planar formats.
Some drivers already have preliminary support for data offsets customization of capture buffers, like NVIDIA Tegra video decoder driver for example, and V4L allows applications to provide data offsets for multi-planar output buffers, let's support such customization for the capture buffers as well.
Signed-off-by Dmitry Osipenko <[email protected]>--- Documentation/userspace-api/media/v4l/buffer.rst | 9 ++++++++- drivers/media/common/videobuf2/videobuf2-v4l2.c | 7 +++++++ 2 files changed, 15 insertions(+), 1 deletion(-)
Quoted 13 lines
Documentation/userspace-api/media/v4l/buffer.rst struct v4l2_plane · line 372
Quoted 10 lines
Documentation/userspace-api/media/v4l/buffer.rst +1
-1@@ -369,13 +369,20 @@ struct v4l2_plane - ``data_offset`` - Offset in bytes to video data in the plane. Drivers must set this field when ``type`` refers to a capture stream, applications when - it refers to an output stream. + it refers to an output or capture stream.There is a clear contradiction in this paragraph. Both the driver and the application MUST set the
data_offset.
I'm not sure where the contradiction is. Application must initialize the data_offset and driver must set data_offset too, if it's appropriate to do that for a particular driver.
Quoted 3 lines
Would it be possible to demo your idea by implementing this in a virtual driver ? vivid already have
data_offsetfor capture in some cases, you could verify if your idea works without any conflict in this scenario.
I actually considered implementing it in the vivid driver, but vivid driver already sets the data_offset to fixed values [1], so I decided that not to change it.
But maybe we actually could extend the vivid driver by accepting data_offset from userspace for the cases where the fixed offset value is zero in the driver.. not sure.
[1]
https://elixir.bootlin.com/linux/latest/source/drivers/media/test-drivers/vivid/vivid-vid-cap.c#L172
I verified my idea using the NVIDIA Tegra video decoder driver, which already takes data_offsets for capture planes [3] and libvdpau-tegra imports DRM dma-bufs into the V4L driver [4][5].
[3]
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/media/platform/nvidia/tegra-vde/v4l2.c#n236
[4]
https://github.com/grate-driver/libvdpau-tegra/blob/master/src/decoder.c#L685
[5]
https://github.com/grate-driver/libvdpau-tegra/blob/master/src/v4l2.c#L359
The plan is to extend RK ISP driver with support of data_offsets next, once we'll agree that this acceptable approach and we don't actually need go back to implementing the new VIDIOC_EXT_QBUF ioctl [6].
[6]
https://patchwork.linuxtv.org/project/linux-media/cover/[email protected]/
This patch solves the problem for userspace when it wants to import buffers into V4L in case of multi-planar formats, but it doesn't cover all other possible cases that may require offsets customization too. On the other hand, it's easy to start accepting data_offset for the capture MPLANES without introducing new UAPIs, so I decided that will be best to start easy with the simplest solution.
Le mercredi 23 mars 2022 à 17:28 +0300, Dmitry Osipenko a écrit :
Quoted 55 lines
Hi Nicolas,
Quoted 30 lines · Nicolas Dufresne
Hi Dmitry,
thanks for giving a second look a this issue.
Le mardi 22 mars 2022 à 16:23 +0300, Dmitry Osipenko a écrit :
Quoted 25 lines
Use data offsets provided by applications for multi-planar capture buffers. This allows V4L to import and use dma-bufs exported by other subsystems in cases where application wants to customize data offsets of capture buffers in order to meet hardware alignment requirements of both dma-buf exporter and importer.
This feature is wanted for providing a better support of media hardware found on Chromebooks. In particular display and camera ISP hardware of Rockchip and MediaTek SoCs require special handling by userspace because display h/w has specific alignment requirements that don't match default alignments expected by V4L and there is a need to customize the data offsets in case of multi-planar formats.
Some drivers already have preliminary support for data offsets customization of capture buffers, like NVIDIA Tegra video decoder driver for example, and V4L allows applications to provide data offsets for multi-planar output buffers, let's support such customization for the capture buffers as well.
Signed-off-by Dmitry Osipenko <[email protected]>--- Documentation/userspace-api/media/v4l/buffer.rst | 9 ++++++++- drivers/media/common/videobuf2/videobuf2-v4l2.c | 7 +++++++ 2 files changed, 15 insertions(+), 1 deletion(-)Quoted 13 lines
Documentation/userspace-api/media/v4l/buffer.rst struct v4l2_plane · line 372
Quoted 10 lines
Documentation/userspace-api/media/v4l/buffer.rst +1
-1@@ -369,13 +369,20 @@ struct v4l2_plane - ``data_offset`` - Offset in bytes to video data in the plane. Drivers must set this field when ``type`` refers to a capture stream, applications when - it refers to an output stream. + it refers to an output or capture stream.There is a clear contradiction in this paragraph. Both the driver and the application MUST set the
data_offset.I'm not sure where the contradiction is. Application must initialize the
data_offsetand driver must setdata_offsettoo, if it's appropriate to do that for a particular driver.Quoted 3 lines
Would it be possible to demo your idea by implementing this in a virtual driver ? vivid already have
data_offsetfor capture in some cases, you could verify if your idea works without any conflict in this scenario.
Quoted 7 lines
I actually considered implementing it in the vivid driver, but vivid driver already sets the
data_offsetto fixed values [1], so I decided that not to change it.But maybe we actually could extend the vivid driver by accepting
data_offsetfrom userspace for the cases where the fixed offset value is zero in the driver.. not sure.
The is the core of the issue, how do you represent both a driver use of data_offset and a userland provided data_offset at the same time. Contradiction might be the wrong term, but minimally there is a large gap in the specification for which I don't have an easy answer.
Quoted 28 lines
[1]
https://elixir.bootlin.com/linux/latest/source/drivers/media/test-drivers/vivid/vivid-vid-cap.c#L172I verified my idea using the NVIDIA Tegra video decoder driver, which already takes
data_offsetsfor capture planes [3] and libvdpau-tegra imports DRM dma-bufs into the V4L driver [4][5].[3]
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/media/platform/nvidia/tegra-vde/v4l2.c#n236
[4]
https://github.com/grate-driver/libvdpau-tegra/blob/master/src/decoder.c#L685
[5]
https://github.com/grate-driver/libvdpau-tegra/blob/master/src/v4l2.c#L359The plan is to extend RK ISP driver with support of
data_offsetsnext, once we'll agree that this acceptable approach and we don't actually need go back to implementing the newVIDIOC_EXT_QBUFioctl [6].[6]
https://patchwork.linuxtv.org/project/linux-media/cover/[email protected]/This patch solves the problem for userspace when it wants to import buffers into V4L in case of multi-planar formats, but it doesn't cover all other possible cases that may require offsets customization too. On the other hand, it's easy to start accepting
data_offsetfor the capture MPLANES without introducing new UAPIs, so I decided that will be best to start easy with the simplest solution.
Quoted 60 lines · Nicolas Dufresne
Le mercredi 23 mars 2022 à 17:28 +0300, Dmitry Osipenko a écrit :
Quoted 59 lines
Hi Nicolas,
Quoted 30 lines · Nicolas Dufresne
Hi Dmitry,
thanks for giving a second look a this issue.
Le mardi 22 mars 2022 à 16:23 +0300, Dmitry Osipenko a écrit :
Quoted 25 lines
Use data offsets provided by applications for multi-planar capture buffers. This allows V4L to import and use dma-bufs exported by other subsystems in cases where application wants to customize data offsets of capture buffers in order to meet hardware alignment requirements of both dma-buf exporter and importer.
This feature is wanted for providing a better support of media hardware found on Chromebooks. In particular display and camera ISP hardware of Rockchip and MediaTek SoCs require special handling by userspace because display h/w has specific alignment requirements that don't match default alignments expected by V4L and there is a need to customize the data offsets in case of multi-planar formats.
Some drivers already have preliminary support for data offsets customization of capture buffers, like NVIDIA Tegra video decoder driver for example, and V4L allows applications to provide data offsets for multi-planar output buffers, let's support such customization for the capture buffers as well.
Signed-off-by Dmitry Osipenko <[email protected]>--- Documentation/userspace-api/media/v4l/buffer.rst | 9 ++++++++- drivers/media/common/videobuf2/videobuf2-v4l2.c | 7 +++++++ 2 files changed, 15 insertions(+), 1 deletion(-)Quoted 13 lines
Documentation/userspace-api/media/v4l/buffer.rst struct v4l2_plane · line 372
Quoted 10 lines
Documentation/userspace-api/media/v4l/buffer.rst +1
-1@@ -369,13 +369,20 @@ struct v4l2_plane - ``data_offset`` - Offset in bytes to video data in the plane. Drivers must set this field when ``type`` refers to a capture stream, applications when - it refers to an output stream. + it refers to an output or capture stream.There is a clear contradiction in this paragraph. Both the driver and the application MUST set the
data_offset.I'm not sure where the contradiction is. Application must initialize the
data_offsetand driver must setdata_offsettoo, if it's appropriate to do that for a particular driver.Quoted 3 lines
Would it be possible to demo your idea by implementing this in a virtual driver ? vivid already have
data_offsetfor capture in some cases, you could verify if your idea works without any conflict in this scenario.I actually considered implementing it in the vivid driver, but vivid driver already sets the
data_offsetto fixed values [1], so I decided that not to change it.
Quoted 8 lines
Quoted 3 lines
But maybe we actually could extend the vivid driver by accepting
data_offsetfrom userspace for the cases where the fixed offset value is zero in the driver.. not sure.The is the core of the issue, how do you represent both a driver use of
data_offsetand a userland provideddata_offsetat the same time. Contradiction might be the wrong term, but minimally there is a large gap in the specification for which I don't have an easy answer.
In the doc-comment I said: "Handling of application's offsets is driver-dependent, application must use the resulting buffer offset."
This can be made stricter using a new V4L2_BUF_FLAG_USE_CAPTURE_MPLANE_DATA_OFFSET, for example. But then it doesn't feel good anymore to me because output buffers don't require a special flag, hence it feels inconsistent.
There is another variant with adding new data_offset fields to the v4l2_plane_pix_format and then will be possible to set data offsets using VIDIOC_S_FMT for both S/MPLANEs, assuming that we won't need more than 3 offsets for SPLANEs. But this is a much more invasive and questionable UAPI extension.
--- >8 ---
@@ -2268,12 +2268,14 @@ struct v4l2_mpeg_vbi_fmt_ivtv {
* this plane will be used * @bytesperline: distance in bytes between the leftmost pixels in two * adjacent lines
+ * @data_offset offset in bytes to the beginning of the
plane's data
* @reserved: drivers and applications must zero this array
*/
struct v4l2_plane_pix_format {
__u32 sizeimage;
__u32 bytesperline;
- __u16 reserved[6];
+ __u32 data_offset[3]; } __attribute__ ((packed));
--- >8 ---
The root of the problem is that DRM UAPI is more flexible and allows to customize offsets for both S/MPLANEs, while V4L doesn't allow to do it at all. I'm exploring all the potential options, so far neither of the proposed variants is ideal.
Le jeudi 24 mars 2022 à 21:20 +0300, Dmitry Osipenko a écrit :
Quoted 4 lines
The root of the problem is that DRM UAPI is more flexible and allows to customize offsets for both S/MPLANEs, while V4L doesn't allow to do it at all. I'm exploring all the potential options, so far neither of the proposed variants is ideal.
In GStreamer kmssink, the way DRM is used, is that if you have 2 planes in your pixel format, but only received 1 DMABuf, we will pass this DMABuf twice (well GEM handles, but twice), with appropriate offset.
With this in mind, the idea for V4L2 could be to always resort to MPLANE for this purpose. The tricky part for userland is that it needs to know the dual pixel format and map that accordingly. That is a bit difficult and this is something Helen was trying to address with the v4l2_buffer_ext (that and allowing space to store DRM Modifiers in the future).
The second challenge is the overhead. In DRM, as we "prime" the DMABuf into handles, this gives a kernel object to store any relevant information about the buffer. So having it duplicate can be done at no cost. In V4L2, the driver would need to handle that more often. Specially that despite the recommendation (except for primary buffer decoder, were this is mandatory), we don't force a strict DMABuf / Buffer IDX mapping.
Nicolas
Hi Nicolas & Dmitry
Quoted 17 lines · Nicolas Dufresne
Quoted 4 lines · Dmitry Osipenko
The root of the problem is that DRM UAPI is more flexible and allows to customize offsets for both S/MPLANEs, while V4L doesn't allow to do it at all. I'm exploring all the potential options, so far neither of the proposed variants is ideal.
In GStreamer kmssink, the way DRM is used, is that if you have 2 planes in your pixel format, but only received 1 DMABuf, we will pass this DMABuf twice (well GEM handles, but twice), with appropriate offset.
With this in mind, the idea for V4L2 could be to always resort to MPLANE for this purpose. The tricky part for userland is that it needs to know the dual pixel format and map that accordingly. That is a bit difficult and this is something Helen was trying to address with the
v4l2_buffer_ext(that and allowing space to store DRM Modifiers in the future).
Quoted 6 lines
The second challenge is the overhead. In DRM, as we "prime" the DMABuf into handles, this gives a kernel object to store any relevant information about the buffer. So having it duplicate can be done at no cost. In V4L2, the driver would need to handle that more often. Specially that despite the recommendation (except for primary buffer decoder, were this is mandatory), we don't force a strict DMABuf / Buffer IDX mapping.
To throw another use case of data offsets into the mix, I'm keeping a watching eye on implementing stereoscopic capture into libcamera where we want to present the same buffer to the ISP twice (once for each eye) with either a vertical or horizontal offset between the two passes.
Adding a data_offset of either a half line or half the frame is the easiest way around that one, although it could potentially be accommodated through the selection API setting a compose region instead.
Dave
Quoted 27 lines · Dave Stevenson
Hi Nicolas & Dmitry
Quoted 17 lines · Nicolas Dufresne
Quoted 4 lines · Dmitry Osipenko
The root of the problem is that DRM UAPI is more flexible and allows to customize offsets for both S/MPLANEs, while V4L doesn't allow to do it at all. I'm exploring all the potential options, so far neither of the proposed variants is ideal.
In GStreamer kmssink, the way DRM is used, is that if you have 2 planes in your pixel format, but only received 1 DMABuf, we will pass this DMABuf twice (well GEM handles, but twice), with appropriate offset.
With this in mind, the idea for V4L2 could be to always resort to MPLANE for this purpose. The tricky part for userland is that it needs to know the dual pixel format and map that accordingly. That is a bit difficult and this is something Helen was trying to address with the
v4l2_buffer_ext(that and allowing space to store DRM Modifiers in the future).Quoted 6 lines
The second challenge is the overhead. In DRM, as we "prime" the DMABuf into handles, this gives a kernel object to store any relevant information about the buffer. So having it duplicate can be done at no cost. In V4L2, the driver would need to handle that more often. Specially that despite the recommendation (except for primary buffer decoder, were this is mandatory), we don't force a strict DMABuf / Buffer IDX mapping.
Quoted 9 lines
To throw another use case of data offsets into the mix, I'm keeping a watching eye on implementing stereoscopic capture into libcamera where we want to present the same buffer to the ISP twice (once for each eye) with either a vertical or horizontal offset between the two passes.
Adding adata_offsetof either a half line or half the frame is the easiest way around that one, although it could potentially be accommodated through the selection API setting a compose region instead.
Hi Dave,
Thank you for the suggestion about the stereoscopic capture! If you'll manage to test this patch and it will work for you, then please feel free to give yours tested-by.
I'll need to do couple extra checks to ensure that we really need this new feature for the Chromebooks, this will take time. But if you'll be able to confirm that this variant of the patch works for yours case with the stereo cams, then we may try to proceed using the current variant right away.
Quoted 10 lines · Nicolas Dufresne
Le jeudi 24 mars 2022 à 21:20 +0300, Dmitry Osipenko a écrit :
Quoted 4 lines
The root of the problem is that DRM UAPI is more flexible and allows to customize offsets for both S/MPLANEs, while V4L doesn't allow to do it at all. I'm exploring all the potential options, so far neither of the proposed variants is ideal.
In GStreamer kmssink, the way DRM is used, is that if you have 2 planes in your pixel format, but only received 1 DMABuf, we will pass this DMABuf twice (well GEM handles, but twice), with appropriate offset.
Quoted 5 lines
With this in mind, the idea for V4L2 could be to always resort to MPLANE for this purpose. The tricky part for userland is that it needs to know the dual pixel format and map that accordingly. That is a bit difficult and this is something Helen was trying to address with the
v4l2_buffer_ext(that and allowing space to store DRM Modifiers in the future).
FYI: here is Helen's last patch series. Since Helen is no longer active in the media subsystem, someone else who is sufficiently motivated would have to take over.
https://patchwork.linuxtv.org/project/linux-media/cover/[email protected]/
I'm not enthusiastic about messing with data_offset: it was - in hindsight - a bad idea.
Regards,
Hans
Quoted 9 lines
The second challenge is the overhead. In DRM, as we "prime" the DMABuf into handles, this gives a kernel object to store any relevant information about the buffer. So having it duplicate can be done at no cost. In V4L2, the driver would need to handle that more often. Specially that despite the recommendation (except for primary buffer decoder, were this is mandatory), we don't force a strict DMABuf / Buffer IDX mapping.
Nicolas
Hello Hans,
Quoted 19 lines · Hans Verkuil
Quoted 10 lines · Nicolas Dufresne
Le jeudi 24 mars 2022 à 21:20 +0300, Dmitry Osipenko a écrit :
Quoted 4 lines
The root of the problem is that DRM UAPI is more flexible and allows to customize offsets for both S/MPLANEs, while V4L doesn't allow to do it at all. I'm exploring all the potential options, so far neither of the proposed variants is ideal.
In GStreamer kmssink, the way DRM is used, is that if you have 2 planes in your pixel format, but only received 1 DMABuf, we will pass this DMABuf twice (well GEM handles, but twice), with appropriate offset.
Quoted 5 lines
With this in mind, the idea for V4L2 could be to always resort to MPLANE for this purpose. The tricky part for userland is that it needs to know the dual pixel format and map that accordingly. That is a bit difficult and this is something Helen was trying to address with the
v4l2_buffer_ext(that and allowing space to store DRM Modifiers in the future).
Quoted 8 lines
FYI: here is Helen's last patch series. Since Helen is no longer active in the media subsystem, someone else who is sufficiently motivated would have to take over.
https://patchwork.linuxtv.org/project/linux-media/cover/[email protected]/
I'm not enthusiastic about messing with
data_offset: it was - in hindsight - a bad idea.
I'm aware of the Helen's work. To me the addition of the new IOCTLs that partially duplicate the older ones doesn't feel like the best approach. But since you're good with it, then I'll try to refresh the Helen's work for 5.20 and we'll see where it will go.
Signature · 2 lines
Best regards, Dmitry
On 5/19/22 13:22, Dmitry Osipenko wrote:
...
Quoted 23 lines
Quoted 23 lines · Nicolas Dufresne
Quoted 10 lines
Le jeudi 24 mars 2022 à 21:20 +0300, Dmitry Osipenko a écrit :
Quoted 4 lines
The root of the problem is that DRM UAPI is more flexible and allows to customize offsets for both S/MPLANEs, while V4L doesn't allow to do it at all. I'm exploring all the potential options, so far neither of the proposed variants is ideal.
In GStreamer kmssink, the way DRM is used, is that if you have 2 planes in your pixel format, but only received 1 DMABuf, we will pass this DMABuf twice (well GEM handles, but twice), with appropriate offset.
Quoted 5 lines
With this in mind, the idea for V4L2 could be to always resort to MPLANE for this purpose. The tricky part for userland is that it needs to know the dual pixel format and map that accordingly. That is a bit difficult and this is something Helen was trying to address with the
v4l2_buffer_ext(that and allowing space to store DRM Modifiers in the future).FYI: here is Helen's last patch series. Since Helen is no longer active in the media subsystem, someone else who is sufficiently motivated would have to take over.
https://patchwork.linuxtv.org/project/linux-media/cover/[email protected]/
Quoted 8 lines
Quoted 2 lines
I'm not enthusiastic about messing with
data_offset: it was - in hindsight - a bad idea.I'm aware of the Helen's work. To me the addition of the new IOCTLs that partially duplicate the older ones doesn't feel like the best approach. But since you're good with it, then I'll try to refresh the Helen's work for 5.20 and we'll see where it will go.
Hello Hans,
I'm having hard time convincing myself that the addition of the new IOCTLs is the right approach for us. Helen was adding the new QBUF IOCTLs that supposed to:
1. Support DRM modifiers.
2. Support per-plane dma-buf offsets.
3. Support new features that "are good to have", like 64bit timestamp and unified multi/single planar formats.
The idea of using DRM modifiers is obsoleted now. Mixing DRM with V4L will be a mess and nobody really needs that feature anymore because there are not that many compressed framebuffer formats needed by V4L drivers.
The dma-buf offsets is the main thing that is really wanted by multiple users and we can support it by extending the current UAPI, which should be the best option IMO. It will be simple to implement and doesn't require much changes form the existing applications.
If we will ever really need to introduce a brand new UAPI, then I think it should be covering more things like deprecating the creation of multiple buffers at once, making buffers independent and reusable, ability to destroy buffer and etc. This will be a lot of work that doesn't worth the effort today since everybody in userspace already adapted to the current state of V4L.
I typed another variant of the patch that extends the struct v4l2_plane with the new buffer offset field and curious what you're thinking about it. It works fine by replacing the data_offset with the new buf_offset in the Tegra decoder driver since it already supports the data_offset which was previously unused because initially I thought that it was about buffer offsets when implemented V4L support for the Tegra driver. If we will agree on this approach then RKISP and MTK should be the next drivers that may get support for the dma-buf offsets.
Or maybe you have good arguments for the addition of new IOCTLs? Other ideas?
Here is the patch in question:
--- >8 ---
Subject: [PATCH] media: videobuf2: Introduce new buf_offset field for dma-buf
memory planes
Add new optional buf_offset field to struct v4l2_plane of the videdev2 UAPI and support it in the V4L2 core. This new field designates the offset from the start of a memory plane's dma-buf.
The primary user of this new field are downstream software stacks that transition to upstream kernel. ChromeOS is one of those stacks and Android might be the other. The downstream kernel fork of ChromeOS re-uses the data_offset field for the dma-buf offsets, but data_offset has a different meaning, and thus, this approach isn't acceptable for the upstream. The reason why it's needed by ChromeOS is the integration with GPU subsystem that uses DRM. In some cases there is disagreement between V4L and DRM drivers on the memory plane offsets, which is difficult to resolve without overriding the offsets in V4L.
Since not all drivers will support the new buf_offset, the new capability flag "supports_mem_plane_offset" is added to the struct vb2_queue. Drivers supporting dma-buf offsets will have to explicitly enable the new capability.
The buf_offset field is meant to be application-specific because there is no generic way for conveying the h/w requirements to userspace in V4L.
Note that only drivers can validate the dma-buf offsets properly because of the different h/w alignment requirements. Hence we can't support the dma-buf offsets handling in the V4L core and it needs to be done within drivers in the buf_prepare() callback of the vb2_ops.
--- drivers/media/common/videobuf2/videobuf2-core.c | 2 ++ drivers/media/common/videobuf2/videobuf2-v4l2.c | 10 ++++++++++ include/media/videobuf2-core.h | 6 ++++++ include/uapi/linux/videodev2.h | 5 ++++- 4 files changed, 22 insertions(+), 1 deletion(-)
drivers/media/common/videobuf2/videobuf2-core.c +2 -0
@@ -1278,6 +1278,7 @@ static int __prepare_dmabuf(struct vb2_buffer *vb) vb->planes[plane].bytesused =0 ; vb->planes[plane].length =0 ; vb->planes[plane].m.fd =0 ; + vb->planes[plane].buf_offset =0 ; vb->planes[plane].data_offset =0 ;/* Acquire each plane's memory */
@@ -1323,6 +1324,7 @@ static int __prepare_dmabuf(struct vb2_buffer *vb) vb->planes[plane].bytesused = planes[plane].bytesused; vb->planes[plane].length = planes[plane].length; vb->planes[plane].m.fd = planes[plane].m.fd; + vb->planes[plane].buf_offset = planes[plane].buf_offset; vb->planes[plane].data_offset = planes[plane].data_offset; }
drivers/media/common/videobuf2/videobuf2-v4l2.c +10 -0
@@ -219,6 +219,13 @@ static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb, struct v4l2_buffer *b b->m.planes[plane].m.fd; planes[plane].length = b->m.planes[plane].length; + planes[plane].buf_offset = + b->m.planes[plane].buf_offset; + +if (planes[plane].buf_offset && !q->supports_mem_plane_offset) { +dprintk (q,1 ,"dma-buf offset unsupported by the queue\n" ); +return -EINVAL ; + } }break ;default :
@@ -294,6 +301,7 @@ static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb, struct v4l2_buffer *bbreak ; } + planes[0 ].buf_offset =0 ; planes[0 ].data_offset =0 ;if (V4L2_TYPE_IS_OUTPUT (b->type)) {if (b->bytesused ==0 )
@@ -528,6 +536,7 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb) pdst->m.userptr = psrc->m.userptr; else if (q->memory == VB2_MEMORY_DMABUF) pdst->m.fd = psrc->m.fd; + pdst->buf_offset = psrc->buf_offset; pdst->data_offset = psrc->data_offset;memset (pdst->reserved,0 ,sizeof (pdst->reserved)); }
@@ -612,6 +621,7 @@ static int __fill_vb2_buffer(struct vb2_buffer *vb, struct vb2_plane *planes) planes[plane].length = vbuf->planes[plane].length; } planes[plane].bytesused = vbuf->planes[plane].bytesused; + planes[plane].buf_offset = vbuf->planes[plane].buf_offset; planes[plane].data_offset = vbuf->planes[plane].data_offset; }return 0 ;
include/media/videobuf2-core.h +6 -0
@@ -171,6 +171,8 @@ struct vb2_mem_ops {* descriptor associated with this plane. * @data_offset: offset in the plane to the start of data; usually 0, * unless there is a header in front of the data. +* @buf_offset: offset in the memory buffer to the start of plane; always 0 +* for non-dmabuf buffers. * * Should contain enough information to be able to cover all the fields * of &struct v4l2_plane at videodev2.h.
@@ -188,6 +190,7 @@ struct vb2_plane {int fd; } m;unsigned int data_offset; +unsigned int buf_offset; };/**
@@ -506,6 +509,8 @@ struct vb2_buf_ops {* ->finish(). * @non_coherent_mem: when set queue will attempt to allocate buffers using * non-coherent memory. +* @supports_mem_plane_offset: when set, a non-zero buf_offset is allowed +* for dma-buf memory planes. * @lock: pointer to a mutex that protects the &struct vb2_queue. The * driver can set this to a mutex to let the v4l2 core serialize * the queuing ioctls. If the driver wants to handle locking
@@ -586,6 +591,7 @@ struct vb2_queue { unsigned int uses_requests:1; unsigned int allow_cache_hints:1; unsigned int non_coherent_mem:1; + unsigned int supports_mem_plane_offset:1;struct mutex *lock;void *owner;
include/uapi/linux/videodev2.h +4 -1
@@ -996,6 +996,8 @@ struct v4l2_requestbuffers {* @m: union of @mem_offset, @userptr and @fd * @data_offset: offset in the plane to the start of data; usually 0, * unless there is a header in front of the data +* @buf_offset offset in the memory buffer to the start of plane; +* always 0 for non-dmabuf buffers * @reserved: drivers and applications must zero this array * * Multi-planar buffers consist of one or more planes, e.g. an YCbCr buffer
@@ -1012,7 +1014,8 @@ struct v4l2_plane {__s32 fd; } m;__u32 data_offset; -__u32 reserved[11 ]; +__u32 buf_offset; +__u32 reserved[7 ]; };/**
Signature · 1 line
2.36.1
Hi all,
How does the community think about this change these days? This unblocks using a frame whose planes are actually in the same dma buffer with multi-planar format in CAPTURE queue whose memory type is DMABUF in decoding.
I agree that the v4l2_buffer_ext API is a right way to support it, but we haven't achieved the new API support for about 5 years, and not sure if we will do in the near future.
Could the community elaborate merging this change?
Best Regards,
-Hiro
Quoted 69 lines · Dmitry Osipenko
Use data offsets provided by applications for multi-planar capture buffers. This allows V4L to import and use dma-bufs exported by other subsystems in cases where application wants to customize data offsets of capture buffers in order to meet hardware alignment requirements of both dma-buf exporter and importer.
This feature is wanted for providing a better support of media hardware found on Chromebooks. In particular display and camera ISP hardware of Rockchip and MediaTek SoCs require special handling by userspace because display h/w has specific alignment requirements that don't match default alignments expected by V4L and there is a need to customize the data offsets in case of multi-planar formats.
Some drivers already have preliminary support for data offsets customization of capture buffers, like NVIDIA Tegra video decoder driver for example, and V4L allows applications to provide data offsets for multi-planar output buffers, let's support such customization for the capture buffers as well.
Signed-off-by Dmitry Osipenko <[email protected]>--- Documentation/userspace-api/media/v4l/buffer.rst | 9 ++++++++- drivers/media/common/videobuf2/videobuf2-v4l2.c | 7 +++++++ 2 files changed, 15 insertions(+), 1 deletion(-)Documentation/userspace-api/media/v4l/buffer.rst +8
-1@@ -369,13 +369,20 @@ struct v4l2_plane - ``data_offset`` - Offset in bytes to video data in the plane. Drivers must set this field when ``type`` refers to a capture stream, applications when - it refers to an output stream. + it refers to an output or capture stream. .. note:: That data_offset is included in ``bytesused``. So the size of the image in the plane is ``bytesused``-``data_offset`` at offset ``data_offset`` from the start of the plane. + + For capture planes data_offset may be specified by applications + and by drivers. Driver may override application's offset or error + out buffer if offset can't be satisfied by hardware. This allows + applications to customize data offsets of imported dma-bufs. + Handling of application's offsets is driver-dependent, application + must use the resulting buffer offset. * - __u32 - ``reserved[11]`` - Reserved for future use. Should be zeroed by drivers anddrivers/media/common/videobuf2/videobuf2-v4l2.c +7
-0@@ -263,6 +263,13 @@ static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb, struct v4l2_buffer *b psrc->bytesused : pdst->length; pdst->data_offset = psrc->data_offset; } + }else { +for (plane =0 ; plane < vb->num_planes; ++plane) { +struct vb2_plane *pdst = &planes[plane]; +struct v4l2_plane *psrc = &b->m.planes[plane]; + + pdst->data_offset = psrc->data_offset; + } } }else {/*
Hi Hiro,
For those who are interested, the original thread is here:
https://lore.kernel.org/linux-media/[email protected]/
Quoted 11 lines · Hirokazu Honda
Hi all,
How does the community think about this change these days? This unblocks using a frame whose planes are actually in the same dma buffer with multi-planar format in CAPTURE queue whose memory type is DMABUF in decoding.
I agree that thev4l2_buffer_extAPI is a right way to support it, but we haven't achieved the new API support for about 5 years, and not sure if we will do in the near future.Could the community elaborate merging this change?
As I mentioned back in 2022, I am very hesitant to fiddle with data_offset. It was a bad idea that should never have been implemented in hindsight.
The reason that the v4l2_buffer_ext API was never pursued is that nobody was sufficiently motivated to keep working on it and keep pushing for it. It is not because it was a bad idea.
I keep hoping that someone will revisit it and start working on it again. The current API is old and awkward to use for multiplanar buffers. And it is missing features that would be nice to add.
Dmitry's last post [1] in the thread back in 2022 was interesting, both his comments on whether the new ioctls would actually be needed, and his proposal for adding a new buf_offset field to struct v4l2_plane.
That said, I still think a new v4l2_buffer_ext struct would be really nice. The old v4l2_buffer is inefficient (complex compat32 and Y2038 handling, almost out of reserved fields, useless timecode struct, awkward to work with multiple planes), so even without adding new fields there is a lot that can be improved here.
The problem with adding a field to v4l2_plane only is that it limits support to multiplanar drivers. It's probably possible to hack it in struct v4l2_buffer as well, but that's just piling more crap into an already crappy struct.
So that's were I am: I'm not opposed to supporting this feature, but I feel that struct v4l2_buffer has reached end-of-life, and that it is time for a modern replacement.
Other may disagree, I'm happy to hear arguments for that.
Regards,
Hans
[1] https://lore.kernel.org/linux-media/[email protected]/
Quoted 74 lines
Best Regards,
-HiroQuoted 69 lines · Dmitry Osipenko
Use data offsets provided by applications for multi-planar capture buffers. This allows V4L to import and use dma-bufs exported by other subsystems in cases where application wants to customize data offsets of capture buffers in order to meet hardware alignment requirements of both dma-buf exporter and importer.
This feature is wanted for providing a better support of media hardware found on Chromebooks. In particular display and camera ISP hardware of Rockchip and MediaTek SoCs require special handling by userspace because display h/w has specific alignment requirements that don't match default alignments expected by V4L and there is a need to customize the data offsets in case of multi-planar formats.
Some drivers already have preliminary support for data offsets customization of capture buffers, like NVIDIA Tegra video decoder driver for example, and V4L allows applications to provide data offsets for multi-planar output buffers, let's support such customization for the capture buffers as well.
Signed-off-by Dmitry Osipenko <[email protected]>--- Documentation/userspace-api/media/v4l/buffer.rst | 9 ++++++++- drivers/media/common/videobuf2/videobuf2-v4l2.c | 7 +++++++ 2 files changed, 15 insertions(+), 1 deletion(-)Documentation/userspace-api/media/v4l/buffer.rst +8
-1@@ -369,13 +369,20 @@ struct v4l2_plane - ``data_offset`` - Offset in bytes to video data in the plane. Drivers must set this field when ``type`` refers to a capture stream, applications when - it refers to an output stream. + it refers to an output or capture stream. .. note:: That data_offset is included in ``bytesused``. So the size of the image in the plane is ``bytesused``-``data_offset`` at offset ``data_offset`` from the start of the plane. + + For capture planes data_offset may be specified by applications + and by drivers. Driver may override application's offset or error + out buffer if offset can't be satisfied by hardware. This allows + applications to customize data offsets of imported dma-bufs. + Handling of application's offsets is driver-dependent, application + must use the resulting buffer offset. * - __u32 - ``reserved[11]`` - Reserved for future use. Should be zeroed by drivers anddrivers/media/common/videobuf2/videobuf2-v4l2.c +7
-0@@ -263,6 +263,13 @@ static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb, struct v4l2_buffer *b psrc->bytesused : pdst->length; pdst->data_offset = psrc->data_offset; } + }else { +for (plane =0 ; plane < vb->num_planes; ++plane) { +struct vb2_plane *pdst = &planes[plane]; +struct v4l2_plane *psrc = &b->m.planes[plane]; + + pdst->data_offset = psrc->data_offset; + } } }else {/*
Thanks Hans for your quick response.
And I apologize for my late reply.
Quoted 3 lines
So that's were I am: I'm not opposed to supporting this feature, but I feel that struct
v4l2_bufferhas reached end-of-life, and that it is time for a modern replacement.
I got it.
I will discuss in my team if I can contribute to v4l2_buffer_ext work or sponsor that work next year or in 2027.
Regards,
-Hiro
Quoted 133 lines · Hans Verkuil
Hi Hiro,
For those who are interested, the original thread is here:
https://lore.kernel.org/linux-media/[email protected]/
Quoted 11 lines · Hirokazu Honda
Hi all,
How does the community think about this change these days? This unblocks using a frame whose planes are actually in the same dma buffer with multi-planar format in CAPTURE queue whose memory type is DMABUF in decoding.
I agree that thev4l2_buffer_extAPI is a right way to support it, but we haven't achieved the new API support for about 5 years, and not sure if we will do in the near future.Could the community elaborate merging this change?
As I mentioned back in 2022, I am very hesitant to fiddle with
data_offset. It was a bad idea that should never have been implemented in hindsight.The reason that the
v4l2_buffer_extAPI was never pursued is that nobody was sufficiently motivated to keep working on it and keep pushing for it. It is not because it was a bad idea.I keep hoping that someone will revisit it and start working on it again. The current API is old and awkward to use for multiplanar buffers. And it is missing features that would be nice to add.
Dmitry's last post [1] in the thread back in 2022 was interesting, both his comments on whether the new ioctls would actually be needed, and his proposal for adding a new
buf_offsetfield to structv4l2_plane.That said, I still think a new
v4l2_buffer_extstruct would be really nice. The oldv4l2_bufferis inefficient (complex compat32 and Y2038 handling, almost out of reserved fields, useless timecode struct, awkward to work with multiple planes), so even without adding new fields there is a lot that can be improved here.The problem with adding a field to
v4l2_planeonly is that it limits support to multiplanar drivers. It's probably possible to hack it in structv4l2_bufferas well, but that's just piling more crap into an already crappy struct.So that's were I am: I'm not opposed to supporting this feature, but I feel that struct
v4l2_bufferhas reached end-of-life, and that it is time for a modern replacement.Other may disagree, I'm happy to hear arguments for that.
Regards,
Hans
[1] https://lore.kernel.org/linux-media/[email protected]/
Quoted 74 lines
Best Regards,
-HiroQuoted 69 lines · Dmitry Osipenko
Use data offsets provided by applications for multi-planar capture buffers. This allows V4L to import and use dma-bufs exported by other subsystems in cases where application wants to customize data offsets of capture buffers in order to meet hardware alignment requirements of both dma-buf exporter and importer.
This feature is wanted for providing a better support of media hardware found on Chromebooks. In particular display and camera ISP hardware of Rockchip and MediaTek SoCs require special handling by userspace because display h/w has specific alignment requirements that don't match default alignments expected by V4L and there is a need to customize the data offsets in case of multi-planar formats.
Some drivers already have preliminary support for data offsets customization of capture buffers, like NVIDIA Tegra video decoder driver for example, and V4L allows applications to provide data offsets for multi-planar output buffers, let's support such customization for the capture buffers as well.
Signed-off-by Dmitry Osipenko <[email protected]>--- Documentation/userspace-api/media/v4l/buffer.rst | 9 ++++++++- drivers/media/common/videobuf2/videobuf2-v4l2.c | 7 +++++++ 2 files changed, 15 insertions(+), 1 deletion(-)Documentation/userspace-api/media/v4l/buffer.rst +8
-1@@ -369,13 +369,20 @@ struct v4l2_plane - ``data_offset`` - Offset in bytes to video data in the plane. Drivers must set this field when ``type`` refers to a capture stream, applications when - it refers to an output stream. + it refers to an output or capture stream. .. note:: That data_offset is included in ``bytesused``. So the size of the image in the plane is ``bytesused``-``data_offset`` at offset ``data_offset`` from the start of the plane. + + For capture planes data_offset may be specified by applications + and by drivers. Driver may override application's offset or error + out buffer if offset can't be satisfied by hardware. This allows + applications to customize data offsets of imported dma-bufs. + Handling of application's offsets is driver-dependent, application + must use the resulting buffer offset. * - __u32 - ``reserved[11]`` - Reserved for future use. Should be zeroed by drivers anddrivers/media/common/videobuf2/videobuf2-v4l2.c +7
-0@@ -263,6 +263,13 @@ static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb, struct v4l2_buffer *b psrc->bytesused : pdst->length; pdst->data_offset = psrc->data_offset; } + }else { +for (plane =0 ; plane < vb->num_planes; ++plane) { +struct vb2_plane *pdst = &planes[plane]; +struct v4l2_plane *psrc = &b->m.planes[plane]; + + pdst->data_offset = psrc->data_offset; + } } }else {/*
Hi Hirokazu, Hans,
Le vendredi 12 décembre 2025 à 15:15 -0800, Hirokazu Honda a écrit :
Quoted 10 lines
Thanks Hans for your quick response.
And I apologize for my late reply.Quoted 3 lines
So that's were I am: I'm not opposed to supporting this feature, but I feel that struct
v4l2_bufferhas reached end-of-life, and that it is time for a modern replacement.I got it.
I will discuss in my team if I can contribute tov4l2_buffer_extwork or sponsor that work next year or in 2027.
I share Hans feeling in regard to trying to force partial support for data_offset into some space left of v4l2_buffer. This brought me thinking that doing that, or even v4l2_buffer_ext as last proposed was not the right solution to modernize the old V4L2 framework. In fact, I believe that v4l2_buffer_ext would simply replicate the MPLANE disaster, leaving another permanent scar in the API. Just to state simply, MPLANE have lead to duplication of every multi- plane pixel formats, solving some plane alignment issue in multi-allocation cases, but leaving behind the common single allocation case.
For me, the most central issue in V4L2 is that the memory allocation/importation is bound to the operation queues. That brings all sort of issues such
- We can't queue twice the same frame
- We can't mix external buffer with device allocated buffer
- All buffers must have the exact same stride
- Application is responsible for caching which memory goes to which v4l2_buffer
- Attempting to import a buffer requires a free spot in the queue
This adds on top of the v4l2_buffer structure limitation we have been targeting so far. With the growth of modern standard API (think Vulkan Video notably), it becomes apparent that the model is too inflexible. This inability to separate memory allocation and importation from operations creates a lot of complexity in user-space, leading to complicated bugs.
I've been quite about it, since until now I didn't have a solution in mind, but I recently come with some ideas. I'll will try develop these ideas, at least in prose for now and come up with an RFC, hopefully somewhere beginning of January 2026. That my proposal is accepted or not isn't quite relevant. But hopefully it will be a starter to go go beyond just fixing what we see. In fact, this next step is for me doing to be quite decisive if I continue doing codecs in V4L2 or not in the long run. But I'm sure this is not just about video codecs.
regards,
Nicolas
Tomasz Figa
maintainerHi everyone,
Quoted 33 lines · Nicolas Dufresne
Hi Hirokazu, Hans,
Quoted 10 lines · Hirokazu Honda
Thanks Hans for your quick response.
And I apologize for my late reply.Quoted 3 lines
So that's were I am: I'm not opposed to supporting this feature, but I feel that struct
v4l2_bufferhas reached end-of-life, and that it is time for a modern replacement.I got it.
I will discuss in my team if I can contribute tov4l2_buffer_extwork or sponsor that work next year or in 2027.I share Hans feeling in regard to trying to force partial support for
data_offsetinto some space left ofv4l2_buffer. This brought me thinking that doing that, or evenv4l2_buffer_extas last proposed was not the right solution to modernize the old V4L2 framework. In fact, I believe thatv4l2_buffer_extwould simply replicate the MPLANE disaster, leaving another permanent scar in the API. Just to state simply, MPLANE have lead to duplication of every multi- plane pixel formats, solving some plane alignment issue in multi-allocation cases, but leaving behind the common single allocation case.For me, the most central issue in V4L2 is that the memory allocation/importation is bound to the operation queues. That brings all sort of issues such
- We can't queue twice the same frame
- We can't mix external buffer with device allocated buffer
- All buffers must have the exact same stride
- Application is responsible for caching which memory goes to whichv4l2_buffer
- Attempting to import a buffer requires a free spot in the queue
Quoted 5 lines
This adds on top of the
v4l2_bufferstructure limitation we have been targeting so far. With the growth of modern standard API (think Vulkan Video notably), it becomes apparent that the model is too inflexible. This inability to separate memory allocation and importation from operations creates a lot of complexity in user-space, leading to complicated bugs.
I more than agree with that.
Quoted 8 lines
I've been quite about it, since until now I didn't have a solution in mind, but I recently come with some ideas. I'll will try develop these ideas, at least in prose for now and come up with an RFC, hopefully somewhere beginning of January 2026. That my proposal is accepted or not isn't quite relevant. But hopefully it will be a starter to go go beyond just fixing what we see. In fact, this next step is for me doing to be quite decisive if I continue doing codecs in V4L2 or not in the long run. But I'm sure this is not just about video codecs.
Looking forward to it. Thanks a lot for looking into it!
Best,
Tomasz
Quoted 23 lines · Nicolas Dufresne
Hi Hirokazu, Hans,
Le vendredi 12 décembre 2025 à 15:15 -0800, Hirokazu Honda a écrit :
Quoted 10 lines
Thanks Hans for your quick response.
And I apologize for my late reply.Quoted 3 lines
So that's were I am: I'm not opposed to supporting this feature, but I feel that struct
v4l2_bufferhas reached end-of-life, and that it is time for a modern replacement.I got it.
I will discuss in my team if I can contribute tov4l2_buffer_extwork or sponsor that work next year or in 2027.I share Hans feeling in regard to trying to force partial support for
data_offsetinto some space left ofv4l2_buffer. This brought me thinking that doing that, or evenv4l2_buffer_extas last proposed was not the right solution to modernize the old V4L2 framework. In fact, I believe thatv4l2_buffer_extwould simply replicate the MPLANE disaster, leaving another permanent scar in the API. Just to state simply, MPLANE have lead to duplication of every multi- plane pixel formats, solving some plane alignment issue in multi-allocation cases, but leaving behind the common single allocation case.
Quoted 6 lines
For me, the most central issue in V4L2 is that the memory allocation/importation is bound to the operation queues. That brings all sort of issues such
- We can't queue twice the same frame
- We can't mix external buffer with device allocated buffer
- All buffers must have the exact same stride
The three limitations above are all technically possible to implement with the current vb2 framework/streaming uAPI, it's just that nobody was ever motivated enough to add support for it.
Quoted 1 line
- Application is responsible for caching which memory goes to which
v4l2_buffer
True, but is this really a big deal?
Quoted 1 line
- Attempting to import a buffer requires a free spot in the queue
True.
Quoted 7 lines
This adds on top of the
v4l2_bufferstructure limitation we have been targeting so far. With the growth of modern standard API (think Vulkan Video notably), it becomes apparent that the model is too inflexible. This inability to separate memory allocation and importation from operations creates a lot of complexity in user-space, leading to complicated bugs.
Quoted 7 lines
I've been quite about it, since until now I didn't have a solution in mind, but I recently come with some ideas. I'll will try develop these ideas, at least in prose for now and come up with an RFC, hopefully somewhere beginning of January 2026. That my proposal is accepted or not isn't quite relevant. But hopefully it will be a starter to go go beyond just fixing what we see. In fact, this next step is for me doing to be quite decisive if I continue doing codecs in V4L2 or not in the long run. But I'm sure this is not just about video codecs.
I'm looking forward to your RFC!
What is important for me is that whatever we come up with, it is something that existing drivers can easily support. A new streaming/buffer allocation uAPI that can only be supported by new drivers will need very, very good reasons for it to be accepted.
The nice thing about the v4l2_buffer_ext proposal was that it can easily be supported by all drivers.
From a technical perspective both struct v4l2_buffer and struct v4l2_format have reached end-of-life: they are full of historical cruft, they are inefficient, the 32-bit/time32 compatibility code is awful and hard to maintain.
Regards,
Hans
Quoted 3 lines
regards,
Nicolas
Thanks Nicolas,
I am also looking forward to your RFC too!
Regards,
-Hiro
Quoted 81 lines · Hans Verkuil
Quoted 23 lines · Nicolas Dufresne
Hi Hirokazu, Hans,
Quoted 10 lines · Hirokazu Honda
Thanks Hans for your quick response.
And I apologize for my late reply.Quoted 3 lines
So that's were I am: I'm not opposed to supporting this feature, but I feel that struct
v4l2_bufferhas reached end-of-life, and that it is time for a modern replacement.I got it.
I will discuss in my team if I can contribute tov4l2_buffer_extwork or sponsor that work next year or in 2027.I share Hans feeling in regard to trying to force partial support for
data_offsetinto some space left ofv4l2_buffer. This brought me thinking that doing that, or evenv4l2_buffer_extas last proposed was not the right solution to modernize the old V4L2 framework. In fact, I believe thatv4l2_buffer_extwould simply replicate the MPLANE disaster, leaving another permanent scar in the API. Just to state simply, MPLANE have lead to duplication of every multi- plane pixel formats, solving some plane alignment issue in multi-allocation cases, but leaving behind the common single allocation case.Quoted 6 lines
For me, the most central issue in V4L2 is that the memory allocation/importation is bound to the operation queues. That brings all sort of issues such
- We can't queue twice the same frame
- We can't mix external buffer with device allocated buffer
- All buffers must have the exact same strideThe three limitations above are all technically possible to implement with the current vb2 framework/streaming uAPI, it's just that nobody was ever motivated enough to add support for it.
Quoted 1 line
- Application is responsible for caching which memory goes to which
v4l2_bufferTrue, but is this really a big deal?
Quoted 1 line
- Attempting to import a buffer requires a free spot in the queue
True.
Quoted 7 lines
This adds on top of the
v4l2_bufferstructure limitation we have been targeting so far. With the growth of modern standard API (think Vulkan Video notably), it becomes apparent that the model is too inflexible. This inability to separate memory allocation and importation from operations creates a lot of complexity in user-space, leading to complicated bugs.Quoted 7 lines
I've been quite about it, since until now I didn't have a solution in mind, but I recently come with some ideas. I'll will try develop these ideas, at least in prose for now and come up with an RFC, hopefully somewhere beginning of January 2026. That my proposal is accepted or not isn't quite relevant. But hopefully it will be a starter to go go beyond just fixing what we see. In fact, this next step is for me doing to be quite decisive if I continue doing codecs in V4L2 or not in the long run. But I'm sure this is not just about video codecs.
I'm looking forward to your RFC!
What is important for me is that whatever we come up with, it is something that existing drivers can easily support. A new streaming/buffer allocation uAPI that can only be supported by new drivers will need very, very good reasons for it to be accepted.
The nice thing about the
v4l2_buffer_extproposal was that it can easily be supported by all drivers.From a technical perspective both struct
v4l2_bufferand structv4l2_formathave reached end-of-life: they are full of historical cruft, they are inefficient, the 32-bit/time32 compatibility code is awful and hard to maintain.Regards,
Hans
Quoted 3 lines
regards,
Nicolas
Hi Hans,
Le mercredi 17 décembre 2025 à 11:02 +0100, Hans Verkuil a écrit :
Quoted 10 lines
Quoted 6 lines
For me, the most central issue in V4L2 is that the memory allocation/importation is bound to the operation queues. That brings all sort of issues such
- We can't queue twice the same frame
- We can't mix external buffer with device allocated buffer
- All buffers must have the exact same strideThe three limitations above are all technically possible to implement with the current vb2 framework/streaming uAPI, it's just that nobody was ever motivated enough to add support for it.
I don't see how technically it is possible without new uAPI to support heterogeneous strides, nor queuing twice the same frame while running in MMAP, or mix device memory and externally imported memory. Please feel free to enlighten me if you have some spare time.
Perhaps worth to mention that this is about doing this without creating glitches or jumps cause by expensive drain and streamoff/on sequences.
Quoted 4 lines
Quoted 2 lines
- Application is responsible for caching which memory goes to which
v4l2_bufferTrue, but is this really a big deal?
Maybe a lesser deal, but its extra complexity for both sides. The current bug being that if you use import mode only to workaround "queuing twice" issue, you will endup with two mapping (which in some stateful codec firmware is not allowed by firmwares). So on top of the lookup userspace is doing to match buffer ids to their memory (have to cache pointers and fd and all), the driver (or vb2) should also implement caching. My proposal imply solving that dual mapping issue for both current and future stream mode.
In userspace, there is also cases, where the video buffers comes from other process, and you don't really know if two FD values points to the same dmabuf. This is the kind of scenarios the DRM subsystem had to deal in compositors, in our case that would be something such as pipewire. This caching is either micro optimization or simply to support firmware limitation, but a guarantee to have 1 memory object for one chunk is in my opinion achievable and allow reducing complexity.
Quoted 27 lines
Quoted 2 lines
- Attempting to import a buffer requires a free spot in the queue
True.
Quoted 7 lines
This adds on top of the
v4l2_bufferstructure limitation we have been targeting so far. With the growth of modern standard API (think Vulkan Video notably), it becomes apparent that the model is too inflexible. This inability to separate memory allocation and importation from operations creates a lot of complexity in user-space, leading to complicated bugs.Quoted 7 lines
I've been quite about it, since until now I didn't have a solution in mind, but I recently come with some ideas. I'll will try develop these ideas, at least in prose for now and come up with an RFC, hopefully somewhere beginning of January 2026. That my proposal is accepted or not isn't quite relevant. But hopefully it will be a starter to go go beyond just fixing what we see. In fact, this next step is for me doing to be quite decisive if I continue doing codecs in V4L2 or not in the long run. But I'm sure this is not just about video codecs.
I'm looking forward to your RFC!
What is important for me is that whatever we come up with, it is something that existing drivers can easily support. A new streaming/buffer allocation uAPI that can only be supported by new drivers will need very, very good reasons for it to be accepted.
Quoted 6 lines
The nice thing about the
v4l2_buffer_extproposal was that it can easily be supported by all drivers.From a technical perspective both struct
v4l2_bufferand structv4l2_formathave reached end-of-life: they are full of historical cruft, they are inefficient, the 32-bit/time32 compatibility code is awful and hard to maintain.
I want to get a lot further in the proposal writing before going into extended debate, but there is absolutely nothing set in stone. I already mention that my proposal it to create traction, and I will have no problem moving direction base on feedback.
I can reassure you that part of the plan is to be able to implement that for all drivers. In fact, I think its a good requirement for any proposal here. My proposal will not replace VB2 or the format. Perhaps few bits of the format will be replicated somewhere else, but not all of it. Perhaps we'll get into breaking down the gigantic "format" structure into manageable part in the future, but this aspect of the _ext proposal, with DRM formats and modifiers is orthogonal to what I have in mind.
Also, the vast majority of what I want to proposal can be implemented inside vb2, without changing any drivers or userspace. But the full gain, I believe, will requires a new v4l2_memory type.
Feel free to think of your own ideas toward these goals, our best bet for the future is if we can combine best ideas together.
regards,
Nicolas
Tomasz Figa
maintainerQuoted 8 lines · Nicolas Dufresne
Hi Hans,
Quoted 4 lines · Hans Verkuil
Quoted 4 lines
For me, the most central issue in V4L2 is that the memory allocation/importation is bound to the operation queues. That brings all sort of issues such
- We can't queue twice the same frame
I think one could just call QBUF with the same buffer index twice (or more).
Quoted 1 line
Quoted 1 line
Quoted 1 line
- We can't mix external buffer with device allocated buffer
VIDIOC_CREATE_BUFS accepts a memory type, so one could use it to create ranges of indexes for different buffer types.
Quoted 1 line
Quoted 1 line
Quoted 1 line
- All buffers must have the exact same stride
This one is less straightforward because V4L2 currently specifies stride as part of the format, not the buffer. In general, buffers in vb2 are just memory planes; and only specific video operations interpret them as a particular format.
So, this is where we might need a brand new UAPI. Although one could also argue that VIDIOC_S_FMT should only apply to buffers that are about to be queued, so one can just call it before queuing a buffer that has a different stride than previous buffers (worst-case
scenario: before every VIDIOC_QBUF).
Quoted 5 lines
Quoted 4 lines
The three limitations above are all technically possible to implement with the current vb2 framework/streaming uAPI, it's just that nobody was ever motivated enough to add support for it.
Quoted 7 lines
I don't see how technically it is possible without new uAPI to support heterogeneous strides, nor queuing twice the same frame while running in MMAP, or mix device memory and externally imported memory. Please feel free to enlighten me if you have some spare time.
Perhaps worth to mention that this is about doing this without creating glitches or jumps cause by expensive drain and streamoff/on sequences.
Some random ideas above ^ :)
Quoted 14 lines
Quoted 5 lines
Quoted 3 lines
- Application is responsible for caching which memory goes to which
v4l2_bufferTrue, but is this really a big deal?
Maybe a lesser deal, but its extra complexity for both sides. The current bug being that if you use import mode only to workaround "queuing twice" issue, you will endup with two mapping (which in some stateful codec firmware is not allowed by firmwares). So on top of the lookup userspace is doing to match buffer ids to their memory (have to cache pointers and fd and all), the driver (or vb2) should also implement caching. My proposal imply solving that dual mapping issue for both current and future stream mode.
Quoted 7 lines
In userspace, there is also cases, where the video buffers comes from other process, and you don't really know if two FD values points to the same dmabuf. This is the kind of scenarios the DRM subsystem had to deal in compositors, in our case that would be something such as pipewire. This caching is either micro optimization or simply to support firmware limitation, but a guarantee to have 1 memory object for one chunk is in my opinion achievable and allow reducing complexity.
This is actually a big deal because, with DMA-bufs, the video code often only receives file descriptors and no unique buffer identifiers. Consequently, the video code has no reliable way to determine which physical buffer a received file descriptor maps to.
And of course if one doesn't match the buffers with the indexes, they get thrashing of vb2 mappings and a bad performance penalty.
Comparing that to the DRM UAPI, the import (prime fd to handle) operation always returns the same buffer handle if a buffer is imported more than once, which uniquely identifies the physical buffer and can be used for this kind of mapping optimization tricks.
Alternatively, we could fix this in the kernel by separating vb2 mappings from buffer indexes and associating them with DMA-buf attachments instead, but last time I spent some time evaluating that it led into some complex lifetime management issues, e.g. when to destroy such a mapping.
Best regards,
Tomasz