Generic line based metadata support, internal pads
Subsystems (2)
Sakari Ailus
maintainerv2 cover Generic line based metadata support, internal pads
Hi folks,
Here are a few patches to add support generic, line based metadata as well as internal pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.
Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.
Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc. The number of these formats is probably going to be about as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.
Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.
Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).
The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).
These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.
The internal (source) pads are an alternative to source routes [1]. The source routes were not universally liked and I do have to say I like re-using existing interface concepts (pads and everything you can do with pads, including access format, selections etc.) wherever it makes sense, instead of duplicating functionality.
Effectively internal source pads behave mostly just like sink pads, but they describe a flow of data that originates from a sub-device instead of arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable and disable routes from internal source pads to sub-device's source pads. The subdev format IOCTLs are usable, too, so one can find which subdev format is available on given internal source pad.
This set depends on these patches:
<URL:https://lore.kernel.org/linux-media/[email protected]/T/#t>
I've also pushed these here and I'll keep updating the branch:
<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>
Questions and comments are most welcome.
Driver support is to be added, as well as an example.
[1] <URL:https://lore.kernel.org/linux-media/[email protected]/>
since v2:
- Added an example demonstrating the use of internal pads. --- Is the level of detail enough for the purpose?
- Improved documentation.
- Added a macro to tell whether a format is a metadata format. (Documentation could be added.)
- A small ReST syntax fix in the same section.
- Drop leftovers of a patch checking for the INTERNAL_SOURCE flag.
since v1:
- Make the new pad flag just "INTERNAL", requiring either SINK or SOURCE pad flag to accompany it. Removed the union in struct v4l2_subdev_route.
- Add the term "stream" to MC glossary.
- Improved and fixed documentation (according to comments).
- Note these formats are little endian.
- Remove 1X8 from the names of the mbus codes. These formats have generally 8 bits per pixel.
- Fix mbus code numbering (had holes in RFC).
- Add new metadata fields to debug prints.
- Fix a minor documentation build issue.
Sakari Ailus (9):
media: Documentation: Align numbered list
media: mc: Check pad flag validity
media: mc: Add INTERNAL pad flag
media: uapi: v4l: Document source routes
media: uapi: Add generic serial metadata mbus formats
media: uapi: Document which mbus format fields are valid for metadata
media: uapi: Add a macro to tell whether an mbus code is metadata
media: uapi: Add generic 8-bit metadata format definitions
media: v4l: Support line-based metadata capture
.../userspace-api/media/glossary.rst | 6 + .../media/mediactl/media-types.rst | 6 + .../userspace-api/media/v4l/dev-meta.rst | 15 + .../userspace-api/media/v4l/dev-subdev.rst | 144 +++++++- .../userspace-api/media/v4l/meta-formats.rst | 1 + .../media/v4l/metafmt-generic.rst | 331 ++++++++++++++++++ .../media/v4l/subdev-formats.rst | 257 ++++++++++++++ .../media/v4l/vidioc-enum-fmt.rst | 7 + .../media/videodev2.h.rst.exceptions | 1 + drivers/media/mc/mc-entity.c | 17 +- drivers/media/v4l2-core/v4l2-ioctl.c | 13 +- include/uapi/linux/media-bus-format.h | 12 + include/uapi/linux/media.h | 1 + include/uapi/linux/v4l2-mediabus.h | 18 +- include/uapi/linux/videodev2.h | 19 + 15 files changed, 824 insertions(+), 24 deletions(-) create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
Signature · 1 line
2.39.2
Sakari Ailus
maintainerPATCH v2 1/9 media: Documentation: Align numbered list
Align lines for numbered list so that Sphinx produces an uniform output for all list entries.
--- .../userspace-api/media/v4l/dev-subdev.rst | 27 +++++++++---------- 1 file changed, 13 insertions(+), 14 deletions(-)
Documentation/userspace-api/media/v4l/dev-subdev.rst +13 -14
@@ -579,20 +579,19 @@ is started. There are three steps in configuring the streams: -1) Set up links. Connect the pads between sub-devices using the :ref:`Media -Controller API <media_controller>` - -2) Streams. Streams are declared and their routing is configured by -setting the routing table for the sub-device using -:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that -setting the routing table will reset formats and selections in the -sub-device to default values. - -3) Configure formats and selections. Formats and selections of each stream -are configured separately as documented for plain sub-devices in -:ref:`format-propagation`. The stream ID is set to the same stream ID -associated with either sink or source pads of routes configured using the -:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. +1) Set up links. Connect the pads between sub-devices using the + :ref:`Media Controller API <media_controller>` + +2) Streams. Streams are declared and their routing is configured by setting the + routing table for the sub-device using :ref:`VIDIOC_SUBDEV_S_ROUTING + <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that setting the routing table will + reset formats and selections in the sub-device to default values. + +3) Configure formats and selections. Formats and selections of each stream are + configured separately as documented for plain sub-devices in + :ref:`format-propagation`. The stream ID is set to the same stream ID + associated with either sink or source pads of routes configured using the + :ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Multiplexed streams setup example ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Signature · 1 line
2.39.2
Sakari Ailus
maintainerPATCH v2 2/9 media: mc: Check pad flag validity
In reply to Sakari AilusCheck the validity of pad flags on entity init. Exactly one of the flags must be set.
--- drivers/media/mc/mc-entity.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
drivers/media/mc/mc-entity.c +10 -1
@@ -197,6 +197,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,struct media_device *mdev = entity->graph_obj.mdev;struct media_pad *iter;unsigned int i =0 ; +int ret =0 ;if (num_pads >=MEDIA_ENTITY_MAX_PADS )return -E2BIG ;
@@ -210,6 +211,14 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,media_entity_for_each_pad (entity, iter) { iter->entity = entity; iter->index = i++; + +if (hweight32 (iter->flags & (MEDIA_PAD_FL_SINK | +MEDIA_PAD_FL_SOURCE )) + !=1 ) { + ret = -EINVAL ; +break ; + } +if (mdev)media_gobj_create (mdev,MEDIA_GRAPH_PAD , &iter->graph_obj);
@@ -218,7 +227,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,if (mdev)mutex_unlock (&mdev->graph_mutex); -return 0 ; +return ret; }EXPORT_SYMBOL_GPL (media_entity_pads_init);
Signature · 1 line
2.39.2
Sakari Ailus
maintainerPATCH v2 3/9 media: mc: Add INTERNAL pad flag
In reply to Sakari AilusInternal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
Also prevent creating links to pads that have been flagged as internal.
--- Documentation/userspace-api/media/glossary.rst | 6 ++++++ Documentation/userspace-api/media/mediactl/media-types.rst | 6 ++++++ drivers/media/mc/mc-entity.c | 6 +++++- include/uapi/linux/media.h | 1 + 4 files changed, 18 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/glossary.rst +6 -0
@@ -173,6 +173,12 @@ Glossary An integrated circuit that integrates all components of a computer or other electronic systems. +_media-glossary-stream: + Stream + A distinct flow of data (image data or metadata) over a media pipeline + from source to sink. A source may be e.g. an image sensor and a sink + e.g. a memory buffer. + V4L2 API **V4L2 userspace API**
Documentation/userspace-api/media/mediactl/media-types.rst +6 -0
@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements .. _MEDIA-PAD-FL-SINK: .. _MEDIA-PAD-FL-SOURCE: .. _MEDIA-PAD-FL-MUST-CONNECT: +.. _MEDIA-PAD-FL-INTERNAL: .. flat-table:: Media pad flags :header-rows: 0
@@ -382,6 +383,11 @@ Types and flags used to represent the media graph elements when this flag isn't set; the absence of the flag doesn't imply there is none. + * - ``MEDIA_PAD_FL_INTERNAL`` + - The internal flag indicates an internal pad that has no external + connections. Such a pad shall not be connected with a link. The + internal flag indicates that the :ref:``stream + <media-glossary-stream>`` either starts or ends in the entity. One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE`` must be set for every pad.
drivers/media/mc/mc-entity.c +5 -1
@@ -1071,7 +1071,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,for (i =0 ; i < entity->num_pads; i++) {if ((entity->pads[i].flags & - (MEDIA_PAD_FL_SINK |MEDIA_PAD_FL_SOURCE )) != pad_type) + (MEDIA_PAD_FL_SINK |MEDIA_PAD_FL_SOURCE | +MEDIA_PAD_FL_INTERNAL )) != pad_type)continue ;if (entity->pads[i].sig_type == sig_type)
@@ -1094,6 +1095,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,return -EINVAL ;if (WARN_ON (!(source->pads[source_pad].flags &MEDIA_PAD_FL_SOURCE )))return -EINVAL ; +if (WARN_ON (source->pads[source_pad].flags &MEDIA_PAD_FL_SOURCE && + source->pads[source_pad].flags &MEDIA_PAD_FL_INTERNAL )) +return -EINVAL ;if (WARN_ON (!(sink->pads[sink_pad].flags &MEDIA_PAD_FL_SINK )))return -EINVAL ;
include/uapi/linux/media.h +1 -0
@@ -208,6 +208,7 @@ struct media_entity_desc {#define MEDIA_PAD_FL_SINK (1U << 0)#define MEDIA_PAD_FL_SOURCE (1U << 1)#define MEDIA_PAD_FL_MUST_CONNECT (1U << 2) +#define MEDIA_PAD_FL_INTERNAL (1U << 3)struct media_pad_desc {__u32 entity;/* entity ID */
Signature · 1 line
2.39.2
Re: [PATCH v2 3/9] media: mc: Add INTERNAL pad flag
Quoted 69 lines · Sakari Ailus
Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
Also prevent creating links to pads that have been flagged as internal.
Signed-off-by Sakari Ailus <[email protected]>--- Documentation/userspace-api/media/glossary.rst | 6 ++++++ Documentation/userspace-api/media/mediactl/media-types.rst | 6 ++++++ drivers/media/mc/mc-entity.c | 6 +++++- include/uapi/linux/media.h | 1 + 4 files changed, 18 insertions(+), 1 deletion(-)Documentation/userspace-api/media/glossary.rst +6
-0@@ -173,6 +173,12 @@ Glossary An integrated circuit that integrates all components of a computer or other electronic systems. +_media-glossary-stream: + Stream + A distinct flow of data (image data or metadata) over a media pipeline + from source to sink. A source may be e.g. an image sensor and a sink + e.g. a memory buffer. + V4L2 API **V4L2 userspace API**Documentation/userspace-api/media/mediactl/media-types.rst +6
-0@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements .. _MEDIA-PAD-FL-SINK: .. _MEDIA-PAD-FL-SOURCE: .. _MEDIA-PAD-FL-MUST-CONNECT: +.. _MEDIA-PAD-FL-INTERNAL: .. flat-table:: Media pad flags :header-rows: 0@@ -382,6 +383,11 @@ Types and flags used to represent the media graph elements when this flag isn't set; the absence of the flag doesn't imply there is none. + * - ``MEDIA_PAD_FL_INTERNAL`` + - The internal flag indicates an internal pad that has no external + connections. Such a pad shall not be connected with a link. The + internal flag indicates that the :ref:``stream + <media-glossary-stream>`` either starts or ends in the entity. One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE`` must be set for every pad.drivers/media/mc/mc-entity.c +2
-1@@ -1071,7 +1071,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,for (i =0 ; i < entity->num_pads; i++) {if ((entity->pads[i].flags & - (MEDIA_PAD_FL_SINK |MEDIA_PAD_FL_SOURCE )) != pad_type) + (MEDIA_PAD_FL_SINK |MEDIA_PAD_FL_SOURCE | +MEDIA_PAD_FL_INTERNAL )) != pad_type)continue ;if (entity->pads[i].sig_type == sig_type)
drivers/media/mc/mc-entity.c media_create_pad_link(struct media_entity *source, u16 source_pad, · line 1100
Quoted 6 lines
drivers/media/mc/mc-entity.c +3
-0@@ -1094,6 +1095,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,return -EINVAL ;if (WARN_ON (!(source->pads[source_pad].flags &MEDIA_PAD_FL_SOURCE )))return -EINVAL ; +if (WARN_ON (source->pads[source_pad].flags &MEDIA_PAD_FL_SOURCE && + source->pads[source_pad].flags &MEDIA_PAD_FL_INTERNAL )) +return -EINVAL ;
What does this check?
Shouldn't we check here if either sink or source is INTERNAL, then fail?
Tomi
Sakari Ailus
maintainerRe: [PATCH v2 3/9] media: mc: Add INTERNAL pad flag
Moi,
Thanks for the review.
Quoted 72 lines · Tomi Valkeinen
Quoted 62 lines · Sakari Ailus
Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
Also prevent creating links to pads that have been flagged as internal.
Signed-off-by Sakari Ailus <[email protected]>--- Documentation/userspace-api/media/glossary.rst | 6 ++++++ Documentation/userspace-api/media/mediactl/media-types.rst | 6 ++++++ drivers/media/mc/mc-entity.c | 6 +++++- include/uapi/linux/media.h | 1 + 4 files changed, 18 insertions(+), 1 deletion(-)Documentation/userspace-api/media/glossary.rst +6
-0@@ -173,6 +173,12 @@ Glossary An integrated circuit that integrates all components of a computer or other electronic systems. +_media-glossary-stream: + Stream + A distinct flow of data (image data or metadata) over a media pipeline + from source to sink. A source may be e.g. an image sensor and a sink + e.g. a memory buffer. + V4L2 API **V4L2 userspace API**Documentation/userspace-api/media/mediactl/media-types.rst +6
-0@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements .. _MEDIA-PAD-FL-SINK: .. _MEDIA-PAD-FL-SOURCE: .. _MEDIA-PAD-FL-MUST-CONNECT: +.. _MEDIA-PAD-FL-INTERNAL: .. flat-table:: Media pad flags :header-rows: 0@@ -382,6 +383,11 @@ Types and flags used to represent the media graph elements when this flag isn't set; the absence of the flag doesn't imply there is none. + * - ``MEDIA_PAD_FL_INTERNAL`` + - The internal flag indicates an internal pad that has no external + connections. Such a pad shall not be connected with a link. The + internal flag indicates that the :ref:``stream + <media-glossary-stream>`` either starts or ends in the entity. One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE`` must be set for every pad.drivers/media/mc/mc-entity.c +2
-1@@ -1071,7 +1071,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,for (i =0 ; i < entity->num_pads; i++) {if ((entity->pads[i].flags & - (MEDIA_PAD_FL_SINK |MEDIA_PAD_FL_SOURCE )) != pad_type) + (MEDIA_PAD_FL_SINK |MEDIA_PAD_FL_SOURCE | +MEDIA_PAD_FL_INTERNAL )) != pad_type)continue ;if (entity->pads[i].sig_type == sig_type)drivers/media/mc/mc-entity.c media_create_pad_link(struct media_entity *source, u16 source_pad, · line 1100
Quoted 6 lines
drivers/media/mc/mc-entity.c +3
-0@@ -1094,6 +1095,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,return -EINVAL ;if (WARN_ON (!(source->pads[source_pad].flags &MEDIA_PAD_FL_SOURCE )))return -EINVAL ; +if (WARN_ON (source->pads[source_pad].flags &MEDIA_PAD_FL_SOURCE && + source->pads[source_pad].flags &MEDIA_PAD_FL_INTERNAL )) +return -EINVAL ;What does this check?
Quoted 1 line
Shouldn't we check here if either sink or source is INTERNAL, then fail?
We shouldn't have any source pads that have the internal flag set, that was the intention here (but you can drop the source flag check). Also both pads can't have the internal flag, I'll add that for v3.
Signature · 3 lines
Kind regards, Sakari Ailus
Re: [PATCH v2 3/9] media: mc: Add INTERNAL pad flag
Quoted 77 lines · Sakari Ailus
Moi,
Thanks for the review.
Quoted 72 lines · Tomi Valkeinen
Quoted 62 lines · Sakari Ailus
Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
Also prevent creating links to pads that have been flagged as internal.
Signed-off-by Sakari Ailus <[email protected]>--- Documentation/userspace-api/media/glossary.rst | 6 ++++++ Documentation/userspace-api/media/mediactl/media-types.rst | 6 ++++++ drivers/media/mc/mc-entity.c | 6 +++++- include/uapi/linux/media.h | 1 + 4 files changed, 18 insertions(+), 1 deletion(-)Documentation/userspace-api/media/glossary.rst +6
-0@@ -173,6 +173,12 @@ Glossary An integrated circuit that integrates all components of a computer or other electronic systems. +_media-glossary-stream: + Stream + A distinct flow of data (image data or metadata) over a media pipeline + from source to sink. A source may be e.g. an image sensor and a sink + e.g. a memory buffer. + V4L2 API **V4L2 userspace API**Documentation/userspace-api/media/mediactl/media-types.rst +6
-0@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements .. _MEDIA-PAD-FL-SINK: .. _MEDIA-PAD-FL-SOURCE: .. _MEDIA-PAD-FL-MUST-CONNECT: +.. _MEDIA-PAD-FL-INTERNAL: .. flat-table:: Media pad flags :header-rows: 0@@ -382,6 +383,11 @@ Types and flags used to represent the media graph elements when this flag isn't set; the absence of the flag doesn't imply there is none. + * - ``MEDIA_PAD_FL_INTERNAL`` + - The internal flag indicates an internal pad that has no external + connections. Such a pad shall not be connected with a link. The + internal flag indicates that the :ref:``stream + <media-glossary-stream>`` either starts or ends in the entity. One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE`` must be set for every pad.drivers/media/mc/mc-entity.c +2
-1@@ -1071,7 +1071,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,for (i =0 ; i < entity->num_pads; i++) {if ((entity->pads[i].flags & - (MEDIA_PAD_FL_SINK |MEDIA_PAD_FL_SOURCE )) != pad_type) + (MEDIA_PAD_FL_SINK |MEDIA_PAD_FL_SOURCE | +MEDIA_PAD_FL_INTERNAL )) != pad_type)continue ;if (entity->pads[i].sig_type == sig_type)drivers/media/mc/mc-entity.c media_create_pad_link(struct media_entity *source, u16 source_pad, · line 1100
Quoted 6 lines
drivers/media/mc/mc-entity.c +3
-0@@ -1094,6 +1095,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,return -EINVAL ;if (WARN_ON (!(source->pads[source_pad].flags &MEDIA_PAD_FL_SOURCE )))return -EINVAL ; +if (WARN_ON (source->pads[source_pad].flags &MEDIA_PAD_FL_SOURCE && + source->pads[source_pad].flags &MEDIA_PAD_FL_INTERNAL )) +return -EINVAL ;What does this check?
Quoted 5 lines
Quoted 1 line
Shouldn't we check here if either sink or source is INTERNAL, then fail?
We shouldn't have any source pads that have the internal flag set, that was the intention here (but you can drop the source flag check). Also both pads can't have the internal flag, I'll add that for v3.
The function is media_create_pad_link(), isn't that about creating links between entities? Internal pads shouldn't be allowed on either side.
Tomi
Sakari Ailus
maintainerRe: [PATCH v2 3/9] media: mc: Add INTERNAL pad flag
Quoted 78 lines · Tomi Valkeinen
Quoted 77 lines · Sakari Ailus
Moi,
Thanks for the review.
Quoted 72 lines · Tomi Valkeinen
Quoted 62 lines · Sakari Ailus
Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
Also prevent creating links to pads that have been flagged as internal.
Signed-off-by Sakari Ailus <[email protected]>--- Documentation/userspace-api/media/glossary.rst | 6 ++++++ Documentation/userspace-api/media/mediactl/media-types.rst | 6 ++++++ drivers/media/mc/mc-entity.c | 6 +++++- include/uapi/linux/media.h | 1 + 4 files changed, 18 insertions(+), 1 deletion(-)Documentation/userspace-api/media/glossary.rst +6
-0@@ -173,6 +173,12 @@ Glossary An integrated circuit that integrates all components of a computer or other electronic systems. +_media-glossary-stream: + Stream + A distinct flow of data (image data or metadata) over a media pipeline + from source to sink. A source may be e.g. an image sensor and a sink + e.g. a memory buffer. + V4L2 API **V4L2 userspace API**Documentation/userspace-api/media/mediactl/media-types.rst +6
-0@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements .. _MEDIA-PAD-FL-SINK: .. _MEDIA-PAD-FL-SOURCE: .. _MEDIA-PAD-FL-MUST-CONNECT: +.. _MEDIA-PAD-FL-INTERNAL: .. flat-table:: Media pad flags :header-rows: 0@@ -382,6 +383,11 @@ Types and flags used to represent the media graph elements when this flag isn't set; the absence of the flag doesn't imply there is none. + * - ``MEDIA_PAD_FL_INTERNAL`` + - The internal flag indicates an internal pad that has no external + connections. Such a pad shall not be connected with a link. The + internal flag indicates that the :ref:``stream + <media-glossary-stream>`` either starts or ends in the entity. One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE`` must be set for every pad.drivers/media/mc/mc-entity.c +2
-1@@ -1071,7 +1071,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,for (i =0 ; i < entity->num_pads; i++) {if ((entity->pads[i].flags & - (MEDIA_PAD_FL_SINK |MEDIA_PAD_FL_SOURCE )) != pad_type) + (MEDIA_PAD_FL_SINK |MEDIA_PAD_FL_SOURCE | +MEDIA_PAD_FL_INTERNAL )) != pad_type)continue ;if (entity->pads[i].sig_type == sig_type)drivers/media/mc/mc-entity.c media_create_pad_link(struct media_entity *source, u16 source_pad, · line 1100
Quoted 6 lines
drivers/media/mc/mc-entity.c +3
-0@@ -1094,6 +1095,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,return -EINVAL ;if (WARN_ON (!(source->pads[source_pad].flags &MEDIA_PAD_FL_SOURCE )))return -EINVAL ; +if (WARN_ON (source->pads[source_pad].flags &MEDIA_PAD_FL_SOURCE && + source->pads[source_pad].flags &MEDIA_PAD_FL_INTERNAL )) +return -EINVAL ;What does this check?
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Shouldn't we check here if either sink or source is INTERNAL, then fail?
We shouldn't have any source pads that have the internal flag set, that was the intention here (but you can drop the source flag check). Also both pads can't have the internal flag, I'll add that for v3.
The function is
media_create_pad_link(), isn't that about creating links between entities? Internal pads shouldn't be allowed on either side.
Uh, yes, indeed. I'll address this in v3.
Signature · 1 line
Sakari Ailus
Sakari Ailus
maintainerPATCH v2 4/9 media: uapi: v4l: Document source routes
In reply to Sakari AilusDocument how internal pads are used on source routes.
--- .../userspace-api/media/v4l/dev-subdev.rst | 117 ++++++++++++++++++ 1 file changed, 117 insertions(+)
Documentation/userspace-api/media/v4l/dev-subdev.rst +117 -0
@@ -551,6 +551,27 @@ A stream at a specific point in the media pipeline is identified by the sub-device and a (pad, stream) pair. For sub-devices that do not support multiplexed streams the 'stream' field is always 0. +.. _v4l2-subdev-source-routes: + +Internal pads and source routes +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Cases where a single sub-device source pad is traversed by multiple streams one +or more of which originate from within the sub-device itself are special as +there is no external sink pad for such routes. In those cases, the sources of +the internally generated streams are represented by internal sink pads, which +are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>` +pad flag set. + +Internal pads have all the properties of an external pad, including formats and +selections. The format in this case is the source format of the stream. An +internal pad always has a single stream only (0). + +/Source routes/ are routes from an internal sink pad to a(n external) source +pad. Generally source routes are not modifiable but they can be activated and +deactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE +<v4l2-subdev-routing-flags>` flag, depending on driver capabilities. + Interaction between routes, streams, formats and selections ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -666,3 +687,99 @@ A common way to accomplish this is to start from the sensors and propagate the configurations along the stream towards the receiver, using :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each stream endpoint in each sub-device. + +Internal pads setup example +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A simple example of a multiplexed stream setup might be as follows: + +- A camera sensor, with one source pad (0) and two internal sink pads (source of + image data (1) and source of embedded data (2)) and two source routes, one + from each internal sink pad towards the (external) source pad. The embedded + data stream needs to be enabled by activating the related route. + +- Multiplexer bridge (Bridge). The bridge has one sink pad, connected to the + sensor (pad 0), and one source pad (pad 1), which outputs two streams. + +- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0), + connected to the bridge, and two source pads (pads 1-2), going to the DMA + engine. The receiver demultiplexes the incoming streams to the source pads. + +- DMA Engines in the SoC (DMA Engine), one for each stream. Each DMA engine is + connected to a single source pad in the receiver. + +The sensor, the bridge and the receiver are modeled as V4L2 sub-devices, +exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are +modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes. + +To configure this pipeline, the userspace must take the following steps: + +1) Set up media links between entities: connect the sensors to the bridge, +bridge to the receiver, and the receiver to the DMA engines. This step does +not differ from normal non-multiplexed media controller setup. + +2) Configure routing + +.. flat-table:: Camera sensor + :header-rows: 1 + + * - Sink Pad/Stream + - Source Pad/Stream + - Routing Flags + - Comments + * - 1/0 + - 0/0 + - V4L2_SUBDEV_ROUTE_FL_ACTIVE + - Pixel data stream from internal sink pad + * - 2/0 + - 0/1 + - V4L2_SUBDEV_ROUTE_FL_ACTIVE + - Metadata stream from internal sink pad + +.. flat-table:: Bridge routing table + :header-rows: 1 + + * - Sink Pad/Stream + - Source Pad/Stream + - Routing Flags + - Comments + * - 0/0 + - 1/0 + - V4L2_SUBDEV_ROUTE_FL_ACTIVE + - Pixel data stream from camera sensor + * - 0/1 + - 1/1 + - V4L2_SUBDEV_ROUTE_FL_ACTIVE + - Metadata stream from camera sensor + +.. flat-table:: Receiver routing table + :header-rows: 1 + + * - Sink Pad/Stream + - Source Pad/Stream + - Routing Flags + - Comments + * - 0/0 + - 1/0 + - V4L2_SUBDEV_ROUTE_FL_ACTIVE + - Pixel data stream from camera sensor + * - 0/1 + - 2/0 + - V4L2_SUBDEV_ROUTE_FL_ACTIVE + - Metadata stream from camera sensor + +3) Configure formats and selections + +After configuring routing, the next step is configuring the formats and +selections for the streams. This is similar to performing this step without +streams, with just one exception: the ``stream`` field needs to be assigned +to the value of the stream ID. + +A common way to accomplish this is to start from the sensors and propagate the +configurations along the stream towards the receiver, +using :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each +stream endpoint in each sub-device. + +The options available in sensor's routing configuration are dictated by hardware +capabilities: typically camera sensors always produce image data stream while +the embedded data stream typically can be either enabled or disabled.
Signature · 1 line
2.39.2
Sakari Ailus
maintainerPATCH v2 5/9 media: uapi: Add generic serial metadata mbus formats
In reply to Sakari AilusAdd generic serial metadata mbus formats. These formats describe data width and packing but not the content itself. The reason for specifying such formats is that the formats as such are fairly device specific but they are still handled by CSI-2 receiver drivers that should not be aware of device specific formats. What makes generic metadata formats possible is that these formats are parsed by software only, after capturing the data to system memory.
--- .../userspace-api/media/v4l/dev-subdev.rst | 8 +- .../media/v4l/subdev-formats.rst | 257 ++++++++++++++++++ include/uapi/linux/media-bus-format.h | 9 + 3 files changed, 270 insertions(+), 4 deletions(-)
Documentation/userspace-api/media/v4l/dev-subdev.rst +4 -4
@@ -639,8 +639,8 @@ modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes. To configure this pipeline, the userspace must take the following steps: 1) Set up media links between entities: connect the sensors to the bridge, -bridge to the receiver, and the receiver to the DMA engines. This step does -not differ from normal non-multiplexed media controller setup. + bridge to the receiver, and the receiver to the DMA engines. This step does + not differ from normal non-multiplexed media controller setup. 2) Configure routing
@@ -715,8 +715,8 @@ modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes. To configure this pipeline, the userspace must take the following steps: 1) Set up media links between entities: connect the sensors to the bridge, -bridge to the receiver, and the receiver to the DMA engines. This step does -not differ from normal non-multiplexed media controller setup. + bridge to the receiver, and the receiver to the DMA engines. This step does + not differ from normal non-multiplexed media controller setup. 2) Configure routing
Documentation/userspace-api/media/v4l/subdev-formats.rst +257 -0
260 lines not on this page · raw message
include/uapi/linux/media-bus-format.h +9 -0
@@ -173,4 +173,13 @@*/ #define MEDIA_BUS_FMT_METADATA_FIXED 0x7001 +/* Generic line based metadata formats for serial buses. Next is 0x8008. */ +#define MEDIA_BUS_FMT_META_8 0x8001 +#define MEDIA_BUS_FMT_META_10 0x8002 +#define MEDIA_BUS_FMT_META_12 0x8003 +#define MEDIA_BUS_FMT_META_14 0x8004 +#define MEDIA_BUS_FMT_META_16 0x8005 +#define MEDIA_BUS_FMT_META_20 0x8006 +#define MEDIA_BUS_FMT_META_24 0x8007 +#endif /* __LINUX_MEDIA_BUS_FORMAT_H */
Signature · 1 line
2.39.2
Sakari Ailus
maintainerPATCH v2 6/9 media: uapi: Document which mbus format fields are valid for metadata
In reply to Sakari AilusNow that metadata mbus formats have been added, it is necessary to define which fields in struct v4l2_mbus_format are applicable to them (not many).
--- include/uapi/linux/v4l2-mediabus.h | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)
include/uapi/linux/v4l2-mediabus.h +12 -6
@@ -19,12 +19,18 @@* @width: image width * @height: image height * @code: data format code (from enum v4l2_mbus_pixelcode) -* @field: used interlacing type (from enum v4l2_field) -* @colorspace: colorspace of the data (from enum v4l2_colorspace) -* @ycbcr_enc: YCbCr encoding of the data (from enum v4l2_ycbcr_encoding) -* @hsv_enc: HSV encoding of the data (from enum v4l2_hsv_encoding) -* @quantization: quantization of the data (from enum v4l2_quantization) -* @xfer_func: transfer function of the data (from enum v4l2_xfer_func) +* @field: used interlacing type (from enum v4l2_field), not applicable +* to metadata mbus codes +* @colorspace: colorspace of the data (from enum v4l2_colorspace), zero on +* metadata mbus codes +* @ycbcr_enc: YCbCr encoding of the data (from enum v4l2_ycbcr_encoding), zero +* on metadata mbus codes +* @hsv_enc: HSV encoding of the data (from enum v4l2_hsv_encoding), zero on +* metadata mbus codes +* @quantization: quantization of the data (from enum v4l2_quantization), zero +* on metadata mbus codes +* @xfer_func: transfer function of the data (from enum v4l2_xfer_func), zero +* on metadata mbus codes * @flags: flags (V4L2_MBUS_FRAMEFMT_*) * @reserved: reserved bytes that can be later used */
Signature · 1 line
2.39.2
Sakari Ailus
maintainerPATCH v2 7/9 media: uapi: Add a macro to tell whether an mbus code is metadata
In reply to Sakari AilusAdd a macro to tell whether a given mbus code is metadata.
--- include/uapi/linux/media-bus-format.h | 3 +++ 1 file changed, 3 insertions(+)
include/uapi/linux/media-bus-format.h +3 -0
@@ -182,4 +182,7 @@#define MEDIA_BUS_FMT_META_20 0x8006#define MEDIA_BUS_FMT_META_24 0x8007 +#define MEDIA_BUS_FMT_IS_META (code) \ + ((code) & 0xf000 == 0x7000 || (code) & 0xf000 == 0x8000) +#endif /* __LINUX_MEDIA_BUS_FORMAT_H */
Signature · 1 line
2.39.2
Sakari Ailus
maintainerPATCH v2 8/9 media: uapi: Add generic 8-bit metadata format definitions
In reply to Sakari AilusGeneric 8-bit metadata formats define the in-memory data layout but not the format of the data itself. The reasoning for having such formats is to allow CSI-2 receiver drivers to receive and DMA drivers to write the data to memory without knowing a large number of device specific formats.
These formats may be used only in conjunction of a Media controller pipeline where the internal pad of the source sub-device defines the specific format of the data (using an mbus code).
--- .../userspace-api/media/v4l/meta-formats.rst | 1 + .../media/v4l/metafmt-generic.rst | 331 ++++++++++++++++++ drivers/media/v4l2-core/v4l2-ioctl.c | 8 + include/uapi/linux/videodev2.h | 9 + 4 files changed, 349 insertions(+) create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
Documentation/userspace-api/media/v4l/meta-formats.rst +1 -0
@@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only. metafmt-vsp1-hgo metafmt-vsp1-hgt metafmt-vivid + metafmt-generic
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +331 -0
331 lines not on this page · raw message
drivers/media/v4l2-core/v4l2-ioctl.c +8 -0
@@ -1452,6 +1452,14 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)case V4L2_PIX_FMT_Y210 : descr ="10-bit YUYV Packed" ;break ;case V4L2_PIX_FMT_Y212 : descr ="12-bit YUYV Packed" ;break ;case V4L2_PIX_FMT_Y216 : descr ="16-bit YUYV Packed" ;break ; +case V4L2_META_FMT_GENERIC_8 : descr ="8-bit Generic Metadata" ;break ; +case V4L2_META_FMT_GENERIC_CSI2_10 : descr ="8b Generic Meta, 10b CSI-2" ;break ; +case V4L2_META_FMT_GENERIC_CSI2_12 : descr ="8b Generic Meta, 12b CSI-2" ;break ; +case V4L2_META_FMT_GENERIC_CSI2_14 : descr ="8b Generic Meta, 14b CSI-2" ;break ; +case V4L2_META_FMT_GENERIC_CSI2_16 : descr ="8b Generic Meta, 16b CSI-2" ;break ; +case V4L2_META_FMT_GENERIC_CSI2_20 : descr ="8b Generic Meta, 20b CSI-2" ;break ; +case V4L2_META_FMT_GENERIC_CSI2_24 : descr ="8b Generic Meta, 24b CSI-2" ;break ; +case V4L2_META_FMT_GENERIC_CSI2_2_24 : descr ="2x8b Generic Meta, 24b CSI-2" ;break ;default :/* Compressed formats */
include/uapi/linux/videodev2.h +9 -0
@@ -836,6 +836,15 @@ struct v4l2_pix_format {#define V4L2_META_FMT_RK_ISP1_PARAMS v4l2_fourcc('R', 'K', '1', 'P')/* Rockchip ISP1 3A Parameters */ #define V4L2_META_FMT_RK_ISP1_STAT_3A v4l2_fourcc('R', 'K', '1', 'S')/* Rockchip ISP1 3A Statistics */ +#define V4L2_META_FMT_GENERIC_8 v4l2_fourcc('M', 'E', 'T', '8')/* Generic 8-bit metadata */ +#define V4L2_META_FMT_GENERIC_CSI2_10 v4l2_fourcc('M', 'C', '1', 'A')/* 10-bit CSI-2 packed 8-bit metadata */ +#define V4L2_META_FMT_GENERIC_CSI2_12 v4l2_fourcc('M', 'C', '1', 'C')/* 12-bit CSI-2 packed 8-bit metadata */ +#define V4L2_META_FMT_GENERIC_CSI2_14 v4l2_fourcc('M', 'C', '1', 'E')/* 14-bit CSI-2 packed 8-bit metadata */ +#define V4L2_META_FMT_GENERIC_CSI2_16 v4l2_fourcc('M', 'C', '1', 'G')/* 16-bit CSI-2 packed 8-bit metadata */ +#define V4L2_META_FMT_GENERIC_CSI2_20 v4l2_fourcc('M', 'C', '1', 'K')/* 20-bit CSI-2 packed 8-bit metadata */ +#define V4L2_META_FMT_GENERIC_CSI2_24 v4l2_fourcc('M', 'C', '1', 'O')/* 24-bit CSI-2 packed 8-bit metadata */ +#define V4L2_META_FMT_GENERIC_CSI2_2_24 v4l2_fourcc('M', 'C', '2', 'O')/* 2 bytes of 8-bit metadata, 24-bit CSI-2 packed */ +/* priv field value to indicates that subsequent fields are valid. */ #define V4L2_PIX_FMT_PRIV_MAGIC 0xfeedcafe
Signature · 1 line
2.39.2
Sakari Ailus
maintainerPATCH v2 9/9 media: v4l: Support line-based metadata capture
In reply to Sakari Ailusmany camera sensors, among other devices, transmit embedded data and image data for each CSI-2 frame. This embedded data typically contains register configuration of the sensor that has been used to capture the image data of the same frame.
The embedded data is received by the CSI-2 receiver and has the same properties as the image data, including that it is line based: it has width, height and bytesperline (stride).
Add these fields to struct v4l2_meta_format and document them.
Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is line-based i.e. these fields of struct v4l2_meta_format are valid for it.
--- .../userspace-api/media/v4l/dev-meta.rst | 15 +++++++++++++++ .../userspace-api/media/v4l/vidioc-enum-fmt.rst | 7 +++++++ .../media/videodev2.h.rst.exceptions | 1 + drivers/media/v4l2-core/v4l2-ioctl.c | 5 +++-- include/uapi/linux/videodev2.h | 10 ++++++++++ 5 files changed, 36 insertions(+), 2 deletions(-)
Documentation/userspace-api/media/v4l/dev-meta.rst +15 -0
@@ -65,3 +65,18 @@ to 0. - ``buffersize`` - Maximum buffer size in bytes required for data. The value is set by the driver. + * - __u32 + - ``width`` + - Width of a line of metadata in samples. Valid when :c:type`v4l2_fmtdesc` + flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See + :c:func:`VIDIOC_ENUM_FMT`. + * - __u32 + - ``height`` + - Number of rows of metadata. Valid when :c:type`v4l2_fmtdesc` flag + ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See + :c:func:`VIDIOC_ENUM_FMT`. + * - __u32 + - ``bytesperline`` + - Offset in bytes between the beginning of two consecutive lines. Valid + when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is + set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +7 -0
@@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently: The application can ask to configure the quantization of the capture device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with :ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set. + * - ``V4L2_FMT_FLAG_META_LINE_BASED`` + - 0x0200 + - The metadata format is line-based. In this case the ``width``, + ``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are + valid. The buffer consists of ``height`` lines, each having ``width`` + bytes of data and offset between the beginning of each two consecutive + lines is ``bytesperline``. Return Value ============
Documentation/userspace-api/media/videodev2.h.rst.exceptions +1 -0
@@ -215,6 +215,7 @@ replace define V4L2_FMT_FLAG_CSC_XFER_FUNC fmtdesc-flags replace define V4L2_FMT_FLAG_CSC_YCBCR_ENC fmtdesc-flags replace define V4L2_FMT_FLAG_CSC_HSV_ENC fmtdesc-flags replace define V4L2_FMT_FLAG_CSC_QUANTIZATION fmtdesc-flags +replace define V4L2_FMT_FLAG_META_LINE_BASED fmtdesc-flags # V4L2 timecode types replace define V4L2_TC_TYPE_24FPS timecode-type
drivers/media/v4l2-core/v4l2-ioctl.c +3 -2
@@ -343,8 +343,9 @@ static void v4l_print_format(const void *arg, bool write_only)case V4L2_BUF_TYPE_META_OUTPUT : meta = &p->fmt.meta; pixelformat = meta->dataformat; -pr_cont (", dataformat=%p4cc, buffersize=%u\n" , - &pixelformat, meta->buffersize); +pr_cont (", dataformat=%p4cc, buffersize=%u, width=%u, height=%u, bytesperline=%u\n" , + &pixelformat, meta->buffersize, meta->width, + meta->height, meta->bytesperline);break ; } }
include/uapi/linux/videodev2.h +10 -0
@@ -875,6 +875,7 @@ struct v4l2_fmtdesc {#define V4L2_FMT_FLAG_CSC_YCBCR_ENC 0x0080#define V4L2_FMT_FLAG_CSC_HSV_ENC V4L2_FMT_FLAG_CSC_YCBCR_ENC#define V4L2_FMT_FLAG_CSC_QUANTIZATION 0x0100 +#define V4L2_FMT_FLAG_META_LINE_BASED 0x0200/* Frame Size and frame rate enumeration */ /*
@@ -2418,10 +2419,19 @@ struct v4l2_sdr_format {* struct v4l2_meta_format - metadata format definition * @dataformat: little endian four character code (fourcc) * @buffersize: maximum size in bytes required for data +* @width: number of bytes of data per line (valid for line based +* formats only, see format documentation) +* @height: number of lines of data per buffer (valid for line based +* formats only) +* @bytesperline: offset between the beginnings of two adjacent lines in +* bytes (valid for line based formats only) */ struct v4l2_meta_format {__u32 dataformat;__u32 buffersize; +__u32 width; +__u32 height; +__u32 bytesperline; } __attribute__ ((packed));/**
Signature · 1 line
2.39.2
Quoted 114 lines · Sakari Ailus
Hi folks,
Here are a few patches to add support generic, line based metadata as well as internal pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.
Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.
Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc. The number of these formats is probably going to be about as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.
Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.
Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).
The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.
The internal (source) pads are an alternative to source routes [1]. The source routes were not universally liked and I do have to say I like re-using existing interface concepts (pads and everything you can do with pads, including access format, selections etc.) wherever it makes sense, instead of duplicating functionality.
Effectively internal source pads behave mostly just like sink pads, but they describe a flow of data that originates from a sub-device instead of arriving to a sub-device. The
SUBDEV_S_ROUTINGIOCTLs are used to enable and disable routes from internal source pads to sub-device's source pads. The subdev format IOCTLs are usable, too, so one can find which subdev format is available on given internal source pad.This set depends on these patches:
<URL:https://lore.kernel.org/linux-media/[email protected]/T/#t>
I've also pushed these here and I'll keep updating the branch:
<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>
Questions and comments are most welcome.
Driver support is to be added, as well as an example.
[1] <URL:https://lore.kernel.org/linux-media/[email protected]/>
since v2:
- Added an example demonstrating the use of internal pads. --- Is the level of detail enough for the purpose?
- Improved documentation.
- Added a macro to tell whether a format is a metadata format. (Documentation could be added.)
- A small ReST syntax fix in the same section.
- Drop leftovers of a patch checking for the
INTERNAL_SOURCEflag.since v1:
- Make the new pad flag just "INTERNAL", requiring either SINK or SOURCE pad flag to accompany it. Removed the union in struct
v4l2_subdev_route.- Add the term "stream" to MC glossary.
- Improved and fixed documentation (according to comments).
- Note these formats are little endian.
- Remove 1X8 from the names of the mbus codes. These formats have generally 8 bits per pixel.
- Fix mbus code numbering (had holes in RFC).
- Add new metadata fields to debug prints.
- Fix a minor documentation build issue.
Quoted 10 lines
Sakari Ailus (9):
media: Documentation: Align numbered list
media: mc: Check pad flag validity
media: mc: Add INTERNAL pad flag
media: uapi: v4l: Document source routes
media: uapi: Add generic serial metadata mbus formats
media: uapi: Document which mbus format fields are valid for metadata
media: uapi: Add a macro to tell whether an mbus code is metadata
media: uapi: Add generic 8-bit metadata format definitions
media: v4l: Support line-based metadata capture
FYI: building the documentation results in these warnings:
Documentation/media/userspace-api/glossary.rst:117: WARNING: term not in glossary: 'V4L2 API' Documentation/media/userspace-api/glossary.rst:130: WARNING: term not in glossary: 'V4L2 Hardware' Documentation/media/userspace-api/glossary.rst:194: WARNING: term not in glossary: 'V4L2 API' Documentation/media/userspace-api/glossary.rst:207: WARNING: term not in glossary: 'V4L2 API' Documentation/media/userspace-api/glossary.rst:207: WARNING: term not in glossary: 'V4L2 Sub-device'
Regards,
Hans
Quoted 19 lines
.../userspace-api/media/glossary.rst | 6 + .../media/mediactl/media-types.rst | 6 + .../userspace-api/media/v4l/dev-meta.rst | 15 + .../userspace-api/media/v4l/dev-subdev.rst | 144 +++++++- .../userspace-api/media/v4l/meta-formats.rst | 1 + .../media/v4l/metafmt-generic.rst | 331 ++++++++++++++++++ .../media/v4l/subdev-formats.rst | 257 ++++++++++++++ .../media/v4l/vidioc-enum-fmt.rst | 7 + .../media/videodev2.h.rst.exceptions | 1 + drivers/media/mc/mc-entity.c | 17 +- drivers/media/v4l2-core/v4l2-ioctl.c | 13 +- include/uapi/linux/media-bus-format.h | 12 + include/uapi/linux/media.h | 1 + include/uapi/linux/v4l2-mediabus.h | 18 +- include/uapi/linux/videodev2.h | 19 + 15 files changed, 824 insertions(+), 24 deletions(-) create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
Sakari Ailus
maintainerHi Hans,
Quoted 126 lines · Hans Verkuil
Quoted 114 lines · Sakari Ailus
Hi folks,
Here are a few patches to add support generic, line based metadata as well as internal pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.
Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.
Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc. The number of these formats is probably going to be about as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.
Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.
Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).
The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.
The internal (source) pads are an alternative to source routes [1]. The source routes were not universally liked and I do have to say I like re-using existing interface concepts (pads and everything you can do with pads, including access format, selections etc.) wherever it makes sense, instead of duplicating functionality.
Effectively internal source pads behave mostly just like sink pads, but they describe a flow of data that originates from a sub-device instead of arriving to a sub-device. The
SUBDEV_S_ROUTINGIOCTLs are used to enable and disable routes from internal source pads to sub-device's source pads. The subdev format IOCTLs are usable, too, so one can find which subdev format is available on given internal source pad.This set depends on these patches:
<URL:https://lore.kernel.org/linux-media/[email protected]/T/#t>
I've also pushed these here and I'll keep updating the branch:
<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>
Questions and comments are most welcome.
Driver support is to be added, as well as an example.
[1] <URL:https://lore.kernel.org/linux-media/[email protected]/>
since v2:
- Added an example demonstrating the use of internal pads. --- Is the level of detail enough for the purpose?
- Improved documentation.
- Added a macro to tell whether a format is a metadata format. (Documentation could be added.)
- A small ReST syntax fix in the same section.
- Drop leftovers of a patch checking for the
INTERNAL_SOURCEflag.since v1:
- Make the new pad flag just "INTERNAL", requiring either SINK or SOURCE pad flag to accompany it. Removed the union in struct
v4l2_subdev_route.- Add the term "stream" to MC glossary.
- Improved and fixed documentation (according to comments).
- Note these formats are little endian.
- Remove 1X8 from the names of the mbus codes. These formats have generally 8 bits per pixel.
- Fix mbus code numbering (had holes in RFC).
- Add new metadata fields to debug prints.
- Fix a minor documentation build issue.
Quoted 10 lines
Sakari Ailus (9):
media: Documentation: Align numbered list
media: mc: Check pad flag validity
media: mc: Add INTERNAL pad flag
media: uapi: v4l: Document source routes
media: uapi: Add generic serial metadata mbus formats
media: uapi: Document which mbus format fields are valid for metadata
media: uapi: Add a macro to tell whether an mbus code is metadata
media: uapi: Add generic 8-bit metadata format definitions
media: v4l: Support line-based metadata capture
Quoted 7 lines
FYI: building the documentation results in these warnings:
Documentation/media/userspace-api/glossary.rst:117: WARNING: term not in glossary: 'V4L2 API'Documentation/media/userspace-api/glossary.rst:130: WARNING: term not in glossary: 'V4L2 Hardware'Documentation/media/userspace-api/glossary.rst:194: WARNING: term not in glossary: 'V4L2 API'Documentation/media/userspace-api/glossary.rst:207: WARNING: term not in glossary: 'V4L2 API'Documentation/media/userspace-api/glossary.rst:207: WARNING: term not in glossary: 'V4L2 Sub-device'
I've seen them, too. I'll post fixes some time soon...
Signature · 1 line
Sakari Ailus