Generic line based metadata support, internal pads
Subsystems (2)
Sakari Ailus
maintainercover Generic line based metadata support, internal pads
Hi folks,
Here are a few patches to add support generic, line based metadata as well as internal source pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.
Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.
Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc. The number of these formats is probably going to be about as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.
Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.
Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).
The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).
These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.
The internal (source) pads are an alternative to source routes [1]. The source routes were not universally liked and I do have to say I like re-using existing interface concepts (pads and everything you can do with pads, including access format, selections etc.) wherever it makes sense, instead of duplicating functionality.
Effectively internal source pads behave mostly just like sink pads, but they describe a flow of data that originates from a sub-device instead of arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable and disable routes from internal source pads to sub-device's source pads. The subdev format IOCTLs are usable, too, so one can find which subdev format is available on given internal source pad.
This set depends on these patches:
<URL:https://lore.kernel.org/linux-media/[email protected]/T/#t>
I've also pushed these here and I'll keep updating the branch:
<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>
Questions and comments are most welcome.
[1] <URL:https://lore.kernel.org/linux-media/[email protected]/>
Sakari Ailus (7):
media: mc: Add INTERNAL_SOURCE pad type flag
media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs
media: uapi: v4l: Document source routes
media: mc: Check pad flag validity
media: uapi: Add generic serial metadata mbus formats
media: uapi: Add generic 8-bit metadata format definitions
media: v4l: Support line-based metadata capture
.../media/mediactl/media-types.rst | 7 + .../userspace-api/media/v4l/dev-meta.rst | 15 + .../userspace-api/media/v4l/dev-subdev.rst | 18 + .../userspace-api/media/v4l/meta-formats.rst | 1 + .../media/v4l/metafmt-generic.rst | 317 ++++++++++++++++++ .../media/v4l/subdev-formats.rst | 257 ++++++++++++++ .../media/v4l/vidioc-enum-fmt.rst | 7 + .../media/v4l/vidioc-subdev-g-routing.rst | 5 + drivers/media/mc/mc-entity.c | 20 +- drivers/media/v4l2-core/v4l2-ioctl.c | 8 + drivers/media/v4l2-core/v4l2-subdev.c | 6 +- include/uapi/linux/media-bus-format.h | 9 + include/uapi/linux/media.h | 1 + include/uapi/linux/v4l2-subdev.h | 6 +- include/uapi/linux/videodev2.h | 19 ++ 15 files changed, 691 insertions(+), 5 deletions(-) create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
Signature · 1 line
2.30.2
Sakari Ailus
maintainerRFC 1/7 media: mc: Add INTERNAL_SOURCE pad type flag
Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
Also prevent creating links to pads that have been flagged as internal.
--- .../userspace-api/media/mediactl/media-types.rst | 7 +++++++ drivers/media/mc/mc-entity.c | 8 +++++++- include/uapi/linux/media.h | 1 + 3 files changed, 15 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/mediactl/media-types.rst +7 -0
@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements .. _MEDIA-PAD-FL-SINK: .. _MEDIA-PAD-FL-SOURCE: .. _MEDIA-PAD-FL-MUST-CONNECT: +.. _MEDIA-PAD-FL-INTERNAL-SOURCE: .. flat-table:: Media pad flags :header-rows: 0
@@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements when this flag isn't set; the absence of the flag doesn't imply there is none. + * - ``MEDIA_PAD_FL_INTERNAL_SOURCE`` + - This flag indicates an internal pad that has no external + connections. Such a pad may not be connected with a link. The internal + flag indicates that the stream either starts or ends in the + entity. For a given pad, the INTERNAL_SOURCE flag may not be set if + either SINK or SOURCE flags is set. One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE`` must be set for every pad.
drivers/media/mc/mc-entity.c +7 -1
@@ -1062,7 +1062,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,for (i =0 ; i < entity->num_pads; i++) {if ((entity->pads[i].flags & - (MEDIA_PAD_FL_SINK |MEDIA_PAD_FL_SOURCE )) != pad_type) + (MEDIA_PAD_FL_SINK |MEDIA_PAD_FL_SOURCE | +MEDIA_PAD_FL_INTERNAL_SOURCE )) != pad_type)continue ;if (entity->pads[i].sig_type == sig_type)
@@ -1087,6 +1088,11 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,return -EINVAL ;if (WARN_ON (!(sink->pads[sink_pad].flags &MEDIA_PAD_FL_SINK )))return -EINVAL ; +if (WARN_ON (sink->pads[sink_pad].flags & +MEDIA_PAD_FL_INTERNAL_SOURCE ) || +WARN_ON (source->pads[source_pad].flags & +MEDIA_PAD_FL_INTERNAL_SOURCE )) +return -EINVAL ; link =media_add_link (&source->links);if (link ==NULL )
include/uapi/linux/media.h +1 -0
@@ -208,6 +208,7 @@ struct media_entity_desc {#define MEDIA_PAD_FL_SINK (1U << 0)#define MEDIA_PAD_FL_SOURCE (1U << 1)#define MEDIA_PAD_FL_MUST_CONNECT (1U << 2) +#define MEDIA_PAD_FL_INTERNAL_SOURCE (1U << 3)struct media_pad_desc {__u32 entity;/* entity ID */
Signature · 1 line
2.30.2
Re: [RFC 1/7] media: mc: Add INTERNAL_SOURCE pad type flag
Quoted 25 lines · Sakari Ailus
Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
Also prevent creating links to pads that have been flagged as internal.
Signed-off-by Sakari Ailus <[email protected]>--- .../userspace-api/media/mediactl/media-types.rst | 7 +++++++ drivers/media/mc/mc-entity.c | 8 +++++++- include/uapi/linux/media.h | 1 + 3 files changed, 15 insertions(+), 1 deletion(-)Documentation/userspace-api/media/mediactl/media-types.rst +1
-0@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements .. _MEDIA-PAD-FL-SINK: .. _MEDIA-PAD-FL-SOURCE: .. _MEDIA-PAD-FL-MUST-CONNECT: +.. _MEDIA-PAD-FL-INTERNAL-SOURCE: .. flat-table:: Media pad flags :header-rows: 0
Documentation/userspace-api/media/mediactl/media-types.rst Types and flags used to represent the media graph elements · line 388
Quoted 6 lines
Documentation/userspace-api/media/mediactl/media-types.rst +3
-0@@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements when this flag isn't set; the absence of the flag doesn't imply there is none. + * - ``MEDIA_PAD_FL_INTERNAL_SOURCE`` + - This flag indicates an internal pad that has no external + connections. Such a pad may not be connected with a link. The internal
"must not"? Or "shall not"?
Tomi
Sakari Ailus
maintainerRe: [RFC 1/7] media: mc: Add INTERNAL_SOURCE pad type flag
Hi Tomi,
Thank you for the review.
Quoted 24 lines · Tomi Valkeinen
Quoted 23 lines · Sakari Ailus
Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
Also prevent creating links to pads that have been flagged as internal.
Signed-off-by Sakari Ailus <[email protected]>--- .../userspace-api/media/mediactl/media-types.rst | 7 +++++++ drivers/media/mc/mc-entity.c | 8 +++++++- include/uapi/linux/media.h | 1 + 3 files changed, 15 insertions(+), 1 deletion(-)Documentation/userspace-api/media/mediactl/media-types.rst +1
-0@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements .. _MEDIA-PAD-FL-SINK: .. _MEDIA-PAD-FL-SOURCE: .. _MEDIA-PAD-FL-MUST-CONNECT: +.. _MEDIA-PAD-FL-INTERNAL-SOURCE: .. flat-table:: Media pad flags :header-rows: 0
Quoted 8 lines
Documentation/userspace-api/media/mediactl/media-types.rst Types and flags used to represent the media graph elements · line 387
Quoted 6 lines
Documentation/userspace-api/media/mediactl/media-types.rst +3
-0@@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements when this flag isn't set; the absence of the flag doesn't imply there is none. + * - ``MEDIA_PAD_FL_INTERNAL_SOURCE`` + - This flag indicates an internal pad that has no external + connections. Such a pad may not be connected with a link. The internal"must not"? Or "shall not"?
I think "may not" is appropriate. I'd be fine with shall, too, albeit it wouldn't change the meaning in practice.
Signature · 1 line
Sakari Ailus
Re: [RFC 1/7] media: mc: Add INTERNAL_SOURCE pad type flag
Quoted 38 lines · Sakari Ailus
Hi Tomi,
Thank you for the review.
Quoted 24 lines · Tomi Valkeinen
Quoted 23 lines · Sakari Ailus
Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
Also prevent creating links to pads that have been flagged as internal.
Signed-off-by Sakari Ailus <[email protected]>--- .../userspace-api/media/mediactl/media-types.rst | 7 +++++++ drivers/media/mc/mc-entity.c | 8 +++++++- include/uapi/linux/media.h | 1 + 3 files changed, 15 insertions(+), 1 deletion(-)Documentation/userspace-api/media/mediactl/media-types.rst +1
-0@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements .. _MEDIA-PAD-FL-SINK: .. _MEDIA-PAD-FL-SOURCE: .. _MEDIA-PAD-FL-MUST-CONNECT: +.. _MEDIA-PAD-FL-INTERNAL-SOURCE: .. flat-table:: Media pad flags :header-rows: 0Quoted 8 lines
Documentation/userspace-api/media/mediactl/media-types.rst Types and flags used to represent the media graph elements · line 387
Quoted 6 lines
Documentation/userspace-api/media/mediactl/media-types.rst +3
-0@@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements when this flag isn't set; the absence of the flag doesn't imply there is none. + * - ``MEDIA_PAD_FL_INTERNAL_SOURCE`` + - This flag indicates an internal pad that has no external + connections. Such a pad may not be connected with a link. The internal"must not"? Or "shall not"?
Quoted 2 lines
I think "may not" is appropriate. I'd be fine with shall, too, albeit it wouldn't change the meaning in practice.
Ok. I don't speak standardize, and I'm not a native English speaker. But to me "may not be connected" sounds like it is possibly not connected, but also that it is possible for it to be connected.
Tomi
Sakari Ailus
maintainerRe: [RFC 1/7] media: mc: Add INTERNAL_SOURCE pad type flag
Quoted 39 lines · Tomi Valkeinen
Quoted 38 lines · Sakari Ailus
Hi Tomi,
Thank you for the review.
Quoted 32 lines · Tomi Valkeinen
Quoted 24 lines · Sakari Ailus
Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
Also prevent creating links to pads that have been flagged as internal.
Signed-off-by Sakari Ailus <[email protected]>--- .../userspace-api/media/mediactl/media-types.rst | 7 +++++++ drivers/media/mc/mc-entity.c | 8 +++++++- include/uapi/linux/media.h | 1 + 3 files changed, 15 insertions(+), 1 deletion(-)Documentation/userspace-api/media/mediactl/media-types.rst +1
-0@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements .. _MEDIA-PAD-FL-SINK: .. _MEDIA-PAD-FL-SOURCE: .. _MEDIA-PAD-FL-MUST-CONNECT: +.. _MEDIA-PAD-FL-INTERNAL-SOURCE: .. flat-table:: Media pad flags :header-rows: 0Documentation/userspace-api/media/mediactl/media-types.rst Types and flags used to represent the media graph elements · line 387
Quoted 5 lines
Documentation/userspace-api/media/mediactl/media-types.rst +3
-0@@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements when this flag isn't set; the absence of the flag doesn't imply there is none. + * - ``MEDIA_PAD_FL_INTERNAL_SOURCE`` + - This flag indicates an internal pad that has no external + connections. Such a pad may not be connected with a link. The internal"must not"? Or "shall not"?
Quoted 6 lines
Quoted 2 lines
I think "may not" is appropriate. I'd be fine with shall, too, albeit it wouldn't change the meaning in practice.
Ok. I don't speak standardize, and I'm not a native English speaker. But to me "may not be connected" sounds like it is possibly not connected, but also that it is possible for it to be connected.
Ah, yes, I think you're actually right. But that is what I thought when writing the text. ;-) Shall we use shall, then?
Signature · 1 line
Sakari Ailus
Laurent Pinchart
maintainerRe: [RFC 1/7] media: mc: Add INTERNAL_SOURCE pad type flag
In reply to Sakari AilusHi Sakari,
Thank you for the patch.
Quoted 25 lines · Sakari Ailus
Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
Also prevent creating links to pads that have been flagged as internal.
Signed-off-by Sakari Ailus <[email protected]>--- .../userspace-api/media/mediactl/media-types.rst | 7 +++++++ drivers/media/mc/mc-entity.c | 8 +++++++- include/uapi/linux/media.h | 1 + 3 files changed, 15 insertions(+), 1 deletion(-)Documentation/userspace-api/media/mediactl/media-types.rst +1
-0@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements .. _MEDIA-PAD-FL-SINK: .. _MEDIA-PAD-FL-SOURCE: .. _MEDIA-PAD-FL-MUST-CONNECT: +.. _MEDIA-PAD-FL-INTERNAL-SOURCE: .. flat-table:: Media pad flags :header-rows: 0
Documentation/userspace-api/media/mediactl/media-types.rst Types and flags used to represent the media graph elements · line 389
Quoted 7 lines
Documentation/userspace-api/media/mediactl/media-types.rst +4
-0@@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements when this flag isn't set; the absence of the flag doesn't imply there is none. + * - ``MEDIA_PAD_FL_INTERNAL_SOURCE`` + - This flag indicates an internal pad that has no external + connections. Such a pad may not be connected with a link. The internal + flag indicates that the stream either starts or ends in the
There's no mention of "stream" (with this meaning) anywhere in the MC API documentation, so you will need to explain streams first.
Apart from that, and addressing Tomi's comment, this patch seems OK.
Quoted 43 lines
Documentation/userspace-api/media/mediactl/media-types.rst +2
-0+ entity. For a given pad, the INTERNAL_SOURCE flag may not be set if + either SINK or SOURCE flags is set. One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE`` must be set for every pad.drivers/media/mc/mc-entity.c +7
-1@@ -1062,7 +1062,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,for (i =0 ; i < entity->num_pads; i++) {if ((entity->pads[i].flags & - (MEDIA_PAD_FL_SINK |MEDIA_PAD_FL_SOURCE )) != pad_type) + (MEDIA_PAD_FL_SINK |MEDIA_PAD_FL_SOURCE | +MEDIA_PAD_FL_INTERNAL_SOURCE )) != pad_type)continue ;if (entity->pads[i].sig_type == sig_type)@@ -1087,6 +1088,11 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,return -EINVAL ;if (WARN_ON (!(sink->pads[sink_pad].flags &MEDIA_PAD_FL_SINK )))return -EINVAL ; +if (WARN_ON (sink->pads[sink_pad].flags & +MEDIA_PAD_FL_INTERNAL_SOURCE ) || +WARN_ON (source->pads[source_pad].flags & +MEDIA_PAD_FL_INTERNAL_SOURCE )) +return -EINVAL ; link =media_add_link (&source->links);if (link ==NULL )include/uapi/linux/media.h +1
-0@@ -208,6 +208,7 @@ struct media_entity_desc {#define MEDIA_PAD_FL_SINK (1U << 0)#define MEDIA_PAD_FL_SOURCE (1U << 1)#define MEDIA_PAD_FL_MUST_CONNECT (1U << 2) +#define MEDIA_PAD_FL_INTERNAL_SOURCE (1U << 3)struct media_pad_desc {__u32 entity;/* entity ID */
Signature · 3 lines
Regards, Laurent Pinchart
Sakari Ailus
maintainerRe: [RFC 1/7] media: mc: Add INTERNAL_SOURCE pad type flag
Hi Laurent,
Quoted 33 lines · Laurent Pinchart
Hi Sakari,
Thank you for the patch.
Quoted 28 lines · Sakari Ailus
Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
Also prevent creating links to pads that have been flagged as internal.
Signed-off-by Sakari Ailus <[email protected]>--- .../userspace-api/media/mediactl/media-types.rst | 7 +++++++ drivers/media/mc/mc-entity.c | 8 +++++++- include/uapi/linux/media.h | 1 + 3 files changed, 15 insertions(+), 1 deletion(-)Documentation/userspace-api/media/mediactl/media-types.rst +1
-0@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements .. _MEDIA-PAD-FL-SINK: .. _MEDIA-PAD-FL-SOURCE: .. _MEDIA-PAD-FL-MUST-CONNECT: +.. _MEDIA-PAD-FL-INTERNAL-SOURCE: .. flat-table:: Media pad flags :header-rows: 0@@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements when this flag isn't set; the absence of the flag doesn't imply there is none.
Quoted 7 lines
Documentation/userspace-api/media/mediactl/media-types.rst Types and flags used to represent the media graph elements · line 389
Quoted 4 lines
Documentation/userspace-api/media/mediactl/media-types.rst +4
-0+ * - ``MEDIA_PAD_FL_INTERNAL_SOURCE`` + - This flag indicates an internal pad that has no external + connections. Such a pad may not be connected with a link. The internal + flag indicates that the stream either starts or ends in theThere's no mention of "stream" (with this meaning) anywhere in the MC API documentation, so you will need to explain streams first.
I'll add it to glossary and add a reference here.
Quoted 47 lines
Apart from that, and addressing Tomi's comment, this patch seems OK.
Quoted 43 lines
Documentation/userspace-api/media/mediactl/media-types.rst +2
-0+ entity. For a given pad, the INTERNAL_SOURCE flag may not be set if + either SINK or SOURCE flags is set. One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE`` must be set for every pad.drivers/media/mc/mc-entity.c +7
-1@@ -1062,7 +1062,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,for (i =0 ; i < entity->num_pads; i++) {if ((entity->pads[i].flags & - (MEDIA_PAD_FL_SINK |MEDIA_PAD_FL_SOURCE )) != pad_type) + (MEDIA_PAD_FL_SINK |MEDIA_PAD_FL_SOURCE | +MEDIA_PAD_FL_INTERNAL_SOURCE )) != pad_type)continue ;if (entity->pads[i].sig_type == sig_type)@@ -1087,6 +1088,11 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,return -EINVAL ;if (WARN_ON (!(sink->pads[sink_pad].flags &MEDIA_PAD_FL_SINK )))return -EINVAL ; +if (WARN_ON (sink->pads[sink_pad].flags & +MEDIA_PAD_FL_INTERNAL_SOURCE ) || +WARN_ON (source->pads[source_pad].flags & +MEDIA_PAD_FL_INTERNAL_SOURCE )) +return -EINVAL ; link =media_add_link (&source->links);if (link ==NULL )include/uapi/linux/media.h +1
-0@@ -208,6 +208,7 @@ struct media_entity_desc {#define MEDIA_PAD_FL_SINK (1U << 0)#define MEDIA_PAD_FL_SOURCE (1U << 1)#define MEDIA_PAD_FL_MUST_CONNECT (1U << 2) +#define MEDIA_PAD_FL_INTERNAL_SOURCE (1U << 3)struct media_pad_desc {__u32 entity;/* entity ID */
Signature · 3 lines
Regards, Sakari Ailus
Re: [RFC 1/7] media: mc: Add INTERNAL_SOURCE pad type flag
In reply to Sakari AilusQuoted 25 lines · Sakari Ailus
Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
Also prevent creating links to pads that have been flagged as internal.
Signed-off-by Sakari Ailus <[email protected]>--- .../userspace-api/media/mediactl/media-types.rst | 7 +++++++ drivers/media/mc/mc-entity.c | 8 +++++++- include/uapi/linux/media.h | 1 + 3 files changed, 15 insertions(+), 1 deletion(-)Documentation/userspace-api/media/mediactl/media-types.rst +1
-0@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements .. _MEDIA-PAD-FL-SINK: .. _MEDIA-PAD-FL-SOURCE: .. _MEDIA-PAD-FL-MUST-CONNECT: +.. _MEDIA-PAD-FL-INTERNAL-SOURCE: .. flat-table:: Media pad flags :header-rows: 0
Documentation/userspace-api/media/mediactl/media-types.rst Types and flags used to represent the media graph elements · line 387
Quoted 5 lines
Documentation/userspace-api/media/mediactl/media-types.rst +2
-0@@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements when this flag isn't set; the absence of the flag doesn't imply there is none. + * - ``MEDIA_PAD_FL_INTERNAL_SOURCE`` + - This flag indicates an internal pad that has no external
'internal pad' or 'internal source pad'?
Documentation/userspace-api/media/mediactl/media-types.rst Types and flags used to represent the media graph elements
Quoted 3 lines
Documentation/userspace-api/media/mediactl/media-types.rst +3
-0+ connections. Such a pad may not be connected with a link. The internal + flag indicates that the stream either starts or ends in the + entity.
This sentence makes no sense. The flag is for a pad, but then this sentence talks about streams and entities and doesn't relate that to the pad. Also, the flag is INTERNAL_SOURCE, and reading 'stream ends' suggests that it should be INTERNAL_SINK in that case.
I think this needs to be more elaborate and ideally add some examples of the use-case(s) (or refer to an example elsewhere).
For a given pad, the INTERNAL_SOURCE flag may not be set if
Documentation/userspace-api/media/mediactl/media-types.rst Types and flags used to represent the media graph elements · line 393
Quoted 3 lines
Documentation/userspace-api/media/mediactl/media-types.rst +1
-0+ either SINK or SOURCE flags is set. One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
and 'MEDIA_PAD_FL_INTERNAL_SOURCE' as well?
Quoted 32 lines
Documentation/userspace-api/media/mediactl/media-types.rst +0
-0must be set for every pad.drivers/media/mc/mc-entity.c +7
-1@@ -1062,7 +1062,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,for (i =0 ; i < entity->num_pads; i++) {if ((entity->pads[i].flags & - (MEDIA_PAD_FL_SINK |MEDIA_PAD_FL_SOURCE )) != pad_type) + (MEDIA_PAD_FL_SINK |MEDIA_PAD_FL_SOURCE | +MEDIA_PAD_FL_INTERNAL_SOURCE )) != pad_type)continue ;if (entity->pads[i].sig_type == sig_type)@@ -1087,6 +1088,11 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,return -EINVAL ;if (WARN_ON (!(sink->pads[sink_pad].flags &MEDIA_PAD_FL_SINK )))return -EINVAL ; +if (WARN_ON (sink->pads[sink_pad].flags & +MEDIA_PAD_FL_INTERNAL_SOURCE ) || +WARN_ON (source->pads[source_pad].flags & +MEDIA_PAD_FL_INTERNAL_SOURCE )) +return -EINVAL ; link =media_add_link (&source->links);if (link ==NULL )include/uapi/linux/media.h +0
-0
include/uapi/linux/media.h struct media_entity_desc { · line 214
Quoted 7 lines
include/uapi/linux/media.h +1
-0@@ -208,6 +208,7 @@ struct media_entity_desc {#define MEDIA_PAD_FL_SINK (1U << 0)#define MEDIA_PAD_FL_SOURCE (1U << 1)#define MEDIA_PAD_FL_MUST_CONNECT (1U << 2) +#define MEDIA_PAD_FL_INTERNAL_SOURCE (1U << 3)struct media_pad_desc {__u32 entity;/* entity ID */
Regards,
Hans
Sakari Ailus
maintainerRe: [RFC 1/7] media: mc: Add INTERNAL_SOURCE pad type flag
Hi Hans,
Thanks for the review.
Quoted 34 lines · Hans Verkuil
Quoted 25 lines · Sakari Ailus
Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.
Also prevent creating links to pads that have been flagged as internal.
Signed-off-by Sakari Ailus <[email protected]>--- .../userspace-api/media/mediactl/media-types.rst | 7 +++++++ drivers/media/mc/mc-entity.c | 8 +++++++- include/uapi/linux/media.h | 1 + 3 files changed, 15 insertions(+), 1 deletion(-)Documentation/userspace-api/media/mediactl/media-types.rst +1
-0@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements .. _MEDIA-PAD-FL-SINK: .. _MEDIA-PAD-FL-SOURCE: .. _MEDIA-PAD-FL-MUST-CONNECT: +.. _MEDIA-PAD-FL-INTERNAL-SOURCE: .. flat-table:: Media pad flags :header-rows: 0Documentation/userspace-api/media/mediactl/media-types.rst Types and flags used to represent the media graph elements · line 387
Quoted 5 lines
Documentation/userspace-api/media/mediactl/media-types.rst +2
-0@@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements when this flag isn't set; the absence of the flag doesn't imply there is none. + * - ``MEDIA_PAD_FL_INTERNAL_SOURCE`` + - This flag indicates an internal pad that has no external'internal pad' or 'internal source pad'?
Quoted 6 lines
Documentation/userspace-api/media/mediactl/media-types.rst Types and flags used to represent the media graph elements
Quoted 3 lines
Documentation/userspace-api/media/mediactl/media-types.rst +3
-0+ connections. Such a pad may not be connected with a link. The internal + flag indicates that the stream either starts or ends in the + entity.This sentence makes no sense. The flag is for a pad, but then this sentence talks about streams and entities and doesn't relate that to the pad. Also,
Streams are between pads, but are visible on V4L2 only while this is MC related. Documentation-wise this is a bit awkward.
Quoted 2 lines
the flag is
INTERNAL_SOURCE, and reading 'stream ends' suggests that it should beINTERNAL_SINKin that case.
When it comes to routing, it uses fields related to the sink pad, but this is a source.
Based on review of the other patches, it seems both Tomi and Laurent would prefer a plain INTERNAL flag which then would be used in conjunction with the existing SINK flag, to note an internal source. Routing-wise this is a cleaner option as this is associated with the sink related fields in any case.
Quoted 54 lines
I think this needs to be more elaborate and ideally add some examples of the use-case(s) (or refer to an example elsewhere).
For a given pad, the
INTERNAL_SOURCEflag may not be set ifDocumentation/userspace-api/media/mediactl/media-types.rst Types and flags used to represent the media graph elements · line 393
Quoted 3 lines
Documentation/userspace-api/media/mediactl/media-types.rst +1
-0+ either SINK or SOURCE flags is set. One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``and '
MEDIA_PAD_FL_INTERNAL_SOURCE' as well?Quoted 32 lines
Documentation/userspace-api/media/mediactl/media-types.rst +0
-0must be set for every pad.drivers/media/mc/mc-entity.c +7
-1@@ -1062,7 +1062,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,for (i =0 ; i < entity->num_pads; i++) {if ((entity->pads[i].flags & - (MEDIA_PAD_FL_SINK |MEDIA_PAD_FL_SOURCE )) != pad_type) + (MEDIA_PAD_FL_SINK |MEDIA_PAD_FL_SOURCE | +MEDIA_PAD_FL_INTERNAL_SOURCE )) != pad_type)continue ;if (entity->pads[i].sig_type == sig_type)@@ -1087,6 +1088,11 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,return -EINVAL ;if (WARN_ON (!(sink->pads[sink_pad].flags &MEDIA_PAD_FL_SINK )))return -EINVAL ; +if (WARN_ON (sink->pads[sink_pad].flags & +MEDIA_PAD_FL_INTERNAL_SOURCE ) || +WARN_ON (source->pads[source_pad].flags & +MEDIA_PAD_FL_INTERNAL_SOURCE )) +return -EINVAL ; link =media_add_link (&source->links);if (link ==NULL )include/uapi/linux/media.h +0
-0include/uapi/linux/media.h struct media_entity_desc { · line 214
Quoted 7 lines
include/uapi/linux/media.h +1
-0@@ -208,6 +208,7 @@ struct media_entity_desc {#define MEDIA_PAD_FL_SINK (1U << 0)#define MEDIA_PAD_FL_SOURCE (1U << 1)#define MEDIA_PAD_FL_MUST_CONNECT (1U << 2) +#define MEDIA_PAD_FL_INTERNAL_SOURCE (1U << 3)struct media_pad_desc {__u32 entity;/* entity ID */ Regards,
Hans
Signature · 3 lines
Kind regards, Sakari Ailus
Sakari Ailus
maintainerRFC 2/7 media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs
In reply to Sakari AilusTake the new INTERNAL_SOURCE pad flag into account in validating routing IOCTL argument. Effectively this is a SINK pad in this respect. Due to the union there's no need to check for the particular name.
--- drivers/media/v4l2-core/v4l2-subdev.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/media/v4l2-core/v4l2-subdev.c +4 -2
@@ -906,7 +906,8 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,return -EINVAL ;if (!(pads[route->sink_pad].flags & -MEDIA_PAD_FL_SINK )) + (MEDIA_PAD_FL_SINK | +MEDIA_PAD_FL_INTERNAL_SOURCE )))return -EINVAL ;if (route->source_pad >= sd->entity.num_pads)
@@ -1787,7 +1788,8 @@ int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,/* Validate the sink and source pad numbers. */ if (route->sink_pad >= sd->entity.num_pads || - !(sd->entity.pads[route->sink_pad].flags &MEDIA_PAD_FL_SINK )) { + !(sd->entity.pads[route->sink_pad].flags & + (MEDIA_PAD_FL_SINK |MEDIA_PAD_FL_INTERNAL_SOURCE ))) {dev_dbg (sd->dev,"route %u sink (%u) is not a sink pad\n" , i, route->sink_pad); goto out;
Signature · 1 line
2.30.2
Re: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs
Quoted 3 lines · Sakari Ailus
Take the new
INTERNAL_SOURCEpad flag into account in validating routing IOCTL argument. Effectively this is a SINK pad in this respect. Due to the union there's no need to check for the particular name.
What union? The one you add in the next patch?
As a concept the internal source pads sound good, and they work in practice in my tests. But this union is what grates me a bit. We have a flag, MEDIA_PAD_FL_INTERNAL_SOURCE, which tells which field of the union to use, and then we go and do not use the new union field. Well, and also the naming, as we normally have source and sink pads, but now we also have internal source pads, which are actually identical to sink pads...
I understand the idea and reasoning, but the two points above do confuse me and I'm sure would confuse others also.
I wonder if it would be less or more confusing to simplify this by just adding a MEDIA_PAD_FL_INTERNAL, which could be applied to either a source or a sink pad, and would prevent linking to it. The flag would indicate that the stream from/to that pad is generated/consumed internally. (I don't know when you would need an internal pad to consume data, but... who knows, the need might pop up =).
That would mean that an "internal sink pad" would generate a data stream, i.e. it's a "source", but I think a normal sink pad is almost the same anyway: when considering entity's internal routing, the normal sink pad is a "source", and the same logic would apply with the internal pads too.
An internal sink pad that generates a stream is a bit more confusing than an internal source pad, but I think that confusion is less than the rest of the confusion in the code-side that comes with the internal source pads.
Tomi
Quoted 29 lines
Signed-off-by Sakari Ailus <[email protected]>--- drivers/media/v4l2-core/v4l2-subdev.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)drivers/media/v4l2-core/v4l2-subdev.c +4
-2@@ -906,7 +906,8 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,return -EINVAL ;if (!(pads[route->sink_pad].flags & -MEDIA_PAD_FL_SINK )) + (MEDIA_PAD_FL_SINK | +MEDIA_PAD_FL_INTERNAL_SOURCE )))return -EINVAL ;if (route->source_pad >= sd->entity.num_pads)@@ -1787,7 +1788,8 @@ int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,/* Validate the sink and source pad numbers. */ if (route->sink_pad >= sd->entity.num_pads || - !(sd->entity.pads[route->sink_pad].flags &MEDIA_PAD_FL_SINK )) { + !(sd->entity.pads[route->sink_pad].flags & + (MEDIA_PAD_FL_SINK |MEDIA_PAD_FL_INTERNAL_SOURCE ))) {dev_dbg (sd->dev,"route %u sink (%u) is not a sink pad\n" , i, route->sink_pad); goto out;
Sakari Ailus
maintainerRe: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs
Moi,
Quoted 6 lines · Tomi Valkeinen
Quoted 3 lines · Sakari Ailus
Take the new
INTERNAL_SOURCEpad flag into account in validating routing IOCTL argument. Effectively this is a SINK pad in this respect. Due to the union there's no need to check for the particular name.What union? The one you add in the next patch?
Oops. I think we can reorder the patches.
Quoted 7 lines
As a concept the internal source pads sound good, and they work in practice in my tests. But this union is what grates me a bit. We have a flag,
MEDIA_PAD_FL_INTERNAL_SOURCE, which tells which field of the union to use, and then we go and do not use the new union field. Well, and also the naming, as we normally have source and sink pads, but now we also have internal source pads, which are actually identical to sink pads...
The union still should be used by the user space. We're testing flags here and those flags are the same independently of the INTERNAL_SOURCE flag.
I'm fine by not adding that union though, but for the user space I think it's better we have it: explaining that the sink_pad has a different meaning if that flag is set is harder than making it a union member.
Quoted 10 lines
I understand the idea and reasoning, but the two points above do confuse me and I'm sure would confuse others also.
I wonder if it would be less or more confusing to simplify this by just adding a
MEDIA_PAD_FL_INTERNAL, which could be applied to either a source or a sink pad, and would prevent linking to it. The flag would indicate that the stream from/to that pad is generated/consumed internally. (I don't know when you would need an internal pad to consume data, but... who knows, the need might pop up =).
This is another option. But I envision there will be more compatibility issues. Although... generally using such hardware requires knowledge of the device, and we haven't obviously had any support for devices needing this functionality in the tree. So in the end it might not matter much.
Quoted 9 lines
That would mean that an "internal sink pad" would generate a data stream, i.e. it's a "source", but I think a normal sink pad is almost the same
anyway: when considering entity's internal routing, the normal sink pad is a "source", and the same logic would apply with the internal pads too.An internal sink pad that generates a stream is a bit more confusing than an internal source pad, but I think that confusion is less than the rest of the confusion in the code-side that comes with the internal source pads.
I prefer having the possible sources of the confusion in the framework than in the drivers. Therefore I think INTERNAL_SOURCE flag is a (slightly) better option.
I wonder what Llaurent thinks.
Signature · 3 lines
Kind regards, Sakari Ailus
Laurent Pinchart
maintainerRe: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs
Quoted 1 line · Sakari Ailus
Moi,
こんにちは
Quoted 48 lines · Tomi Valkeinen
Quoted 6 lines · Sakari Ailus
Quoted 3 lines
Take the new
INTERNAL_SOURCEpad flag into account in validating routing IOCTL argument. Effectively this is a SINK pad in this respect. Due to the union there's no need to check for the particular name.What union? The one you add in the next patch?
Oops. I think we can reorder the patches.
Quoted 6 lines
As a concept the internal source pads sound good, and they work in practice in my tests. But this union is what grates me a bit. We have a flag,
MEDIA_PAD_FL_INTERNAL_SOURCE, which tells which field of the union to use, and then we go and do not use the new union field. Well, and also the naming, as we normally have source and sink pads, but now we also have internal source pads, which are actually identical to sink pads...The union still should be used by the user space. We're testing flags here and those flags are the same independently of the
INTERNAL_SOURCEflag.I'm fine by not adding that union though, but for the user space I think it's better we have it: explaining that the
sink_padhas a different meaning if that flag is set is harder than making it a union member.Quoted 9 lines
I understand the idea and reasoning, but the two points above do confuse me and I'm sure would confuse others also.
I wonder if it would be less or more confusing to simplify this by just adding a
MEDIA_PAD_FL_INTERNAL, which could be applied to either a source or a sink pad, and would prevent linking to it. The flag would indicate that the stream from/to that pad is generated/consumed internally. (I don't know when you would need an internal pad to consume data, but... who knows, the need might pop up =).This is another option. But I envision there will be more compatibility issues. Although... generally using such hardware requires knowledge of the device, and we haven't obviously had any support for devices needing this functionality in the tree. So in the end it might not matter much.
Quoted 8 lines
That would mean that an "internal sink pad" would generate a data stream, i.e. it's a "source", but I think a normal sink pad is almost the same
anyway: when considering entity's internal routing, the normal sink pad is a "source", and the same logic would apply with the internal pads too.An internal sink pad that generates a stream is a bit more confusing than an internal source pad, but I think that confusion is less than the rest of the confusion in the code-side that comes with the internal source pads.
Quoted 5 lines
I prefer having the possible sources of the confusion in the framework than in the drivers. Therefore I think
INTERNAL_SOURCEflag is a (slightly) better option.I wonder what Llaurent thinks.
I like the idea of a MEDIA_PAD_FL_INTERNAL flag. That's actually how I read patch 1/7, I didn't notice it used MEDIA_PAD_FL_INTERNAL*_SOURCE* :-)
We could define MEDIA_PAD_FL_INTERNAL_SOURCE
#define MEDIA_PAD_FL_INTERNAL_SOURCE (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL)
but I'm not sure it would be less confusing.
Regarding isolating the sources of confusion in the framework rather than spreading them through drivers, I can't disagree, but I think that, for raw camera sensors at least, the best option would be to create a new camera sensor object with a much more tailored API than v4l2_subdev (and of course wrapping that new object in a v4l2_subdev in the framework). This won't address the other types of drivers, but I'm not sure we'll get any in the foreseable future.
Signature · 3 lines
Regards, Laurent Pinchart
Laurent Pinchart
maintainerRe: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs
Quoted 59 lines · Laurent Pinchart
Quoted 1 line · Sakari Ailus
Moi,
こんにちは
Quoted 48 lines · Tomi Valkeinen
Quoted 6 lines · Sakari Ailus
Quoted 3 lines
Take the new
INTERNAL_SOURCEpad flag into account in validating routing IOCTL argument. Effectively this is a SINK pad in this respect. Due to the union there's no need to check for the particular name.What union? The one you add in the next patch?
Oops. I think we can reorder the patches.
Quoted 6 lines
As a concept the internal source pads sound good, and they work in practice in my tests. But this union is what grates me a bit. We have a flag,
MEDIA_PAD_FL_INTERNAL_SOURCE, which tells which field of the union to use, and then we go and do not use the new union field. Well, and also the naming, as we normally have source and sink pads, but now we also have internal source pads, which are actually identical to sink pads...The union still should be used by the user space. We're testing flags here and those flags are the same independently of the
INTERNAL_SOURCEflag.I'm fine by not adding that union though, but for the user space I think it's better we have it: explaining that the
sink_padhas a different meaning if that flag is set is harder than making it a union member.Quoted 9 lines
I understand the idea and reasoning, but the two points above do confuse me and I'm sure would confuse others also.
I wonder if it would be less or more confusing to simplify this by just adding a
MEDIA_PAD_FL_INTERNAL, which could be applied to either a source or a sink pad, and would prevent linking to it. The flag would indicate that the stream from/to that pad is generated/consumed internally. (I don't know when you would need an internal pad to consume data, but... who knows, the need might pop up =).This is another option. But I envision there will be more compatibility issues. Although... generally using such hardware requires knowledge of the device, and we haven't obviously had any support for devices needing this functionality in the tree. So in the end it might not matter much.
Quoted 8 lines
That would mean that an "internal sink pad" would generate a data stream, i.e. it's a "source", but I think a normal sink pad is almost the same
anyway: when considering entity's internal routing, the normal sink pad is a "source", and the same logic would apply with the internal pads too.An internal sink pad that generates a stream is a bit more confusing than an internal source pad, but I think that confusion is less than the rest of the confusion in the code-side that comes with the internal source pads.
Quoted 5 lines
I prefer having the possible sources of the confusion in the framework than in the drivers. Therefore I think
INTERNAL_SOURCEflag is a (slightly) better option.I wonder what Llaurent thinks.
Quoted 7 lines
I like the idea of a
MEDIA_PAD_FL_INTERNALflag. That's actually how I read patch 1/7, I didn't notice it used MEDIA_PAD_FL_INTERNAL*_SOURCE* :-)We could define
MEDIA_PAD_FL_INTERNAL_SOURCE#define
MEDIA_PAD_FL_INTERNAL_SOURCE(MEDIA_PAD_FL_SINK|MEDIA_PAD_FL_INTERNAL)
One option would be to find terms different from sink and source in this case. It would generate less confusion to map (e.g., not a really good name) MEDIA_PAD_FL_INTERNAL_PRODUCER to MEDIA_PAD_FL_SINK and to the sink_pad field than using MEDIA_PAD_FL_INTERNAL_SOURCE.
Quoted 9 lines
but I'm not sure it would be less confusing.
Regarding isolating the sources of confusion in the framework rather than spreading them through drivers, I can't disagree, but I think that, for raw camera sensors at least, the best option would be to create a new camera sensor object with a much more tailored API than
v4l2_subdev(and of course wrapping that new object in av4l2_subdevin the framework). This won't address the other types of drivers, but I'm not sure we'll get any in the foreseable future.
Signature · 3 lines
Regards, Laurent Pinchart
Sakari Ailus
maintainerRe: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs
Heippa Tomi, Laurent,
Quoted 64 lines · Laurent Pinchart
Quoted 63 lines · Laurent Pinchart
Quoted 1 line · Sakari Ailus
Moi,
こんにちは
Quoted 48 lines · Tomi Valkeinen
Quoted 6 lines · Sakari Ailus
Quoted 3 lines
Take the new
INTERNAL_SOURCEpad flag into account in validating routing IOCTL argument. Effectively this is a SINK pad in this respect. Due to the union there's no need to check for the particular name.What union? The one you add in the next patch?
Oops. I think we can reorder the patches.
Quoted 6 lines
As a concept the internal source pads sound good, and they work in practice in my tests. But this union is what grates me a bit. We have a flag,
MEDIA_PAD_FL_INTERNAL_SOURCE, which tells which field of the union to use, and then we go and do not use the new union field. Well, and also the naming, as we normally have source and sink pads, but now we also have internal source pads, which are actually identical to sink pads...The union still should be used by the user space. We're testing flags here and those flags are the same independently of the
INTERNAL_SOURCEflag.I'm fine by not adding that union though, but for the user space I think it's better we have it: explaining that the
sink_padhas a different meaning if that flag is set is harder than making it a union member.Quoted 9 lines
I understand the idea and reasoning, but the two points above do confuse me and I'm sure would confuse others also.
I wonder if it would be less or more confusing to simplify this by just adding a
MEDIA_PAD_FL_INTERNAL, which could be applied to either a source or a sink pad, and would prevent linking to it. The flag would indicate that the stream from/to that pad is generated/consumed internally. (I don't know when you would need an internal pad to consume data, but... who knows, the need might pop up =).This is another option. But I envision there will be more compatibility issues. Although... generally using such hardware requires knowledge of the device, and we haven't obviously had any support for devices needing this functionality in the tree. So in the end it might not matter much.
Quoted 8 lines
That would mean that an "internal sink pad" would generate a data stream, i.e. it's a "source", but I think a normal sink pad is almost the same
anyway: when considering entity's internal routing, the normal sink pad is a "source", and the same logic would apply with the internal pads too.An internal sink pad that generates a stream is a bit more confusing than an internal source pad, but I think that confusion is less than the rest of the confusion in the code-side that comes with the internal source pads.
Quoted 5 lines
I prefer having the possible sources of the confusion in the framework than in the drivers. Therefore I think
INTERNAL_SOURCEflag is a (slightly) better option.I wonder what Llaurent thinks.
I like the idea of a
MEDIA_PAD_FL_INTERNALflag. That's actually how I read patch 1/7, I didn't notice it used MEDIA_PAD_FL_INTERNAL*_SOURCE* :-)
Quoted 8 lines
Quoted 3 lines
We could define
MEDIA_PAD_FL_INTERNAL_SOURCE#define
MEDIA_PAD_FL_INTERNAL_SOURCE(MEDIA_PAD_FL_SINK|MEDIA_PAD_FL_INTERNAL)One option would be to find terms different from sink and source in this case. It would generate less confusion to map (e.g., not a really good name)
MEDIA_PAD_FL_INTERNAL_PRODUCERtoMEDIA_PAD_FL_SINKand to thesink_padfield than usingMEDIA_PAD_FL_INTERNAL_SOURCE.
I don't think this helps as you'd still be accessing the sink pad related fields that are there for another purpose.
Alternatively I'd have the (plain) INTERNAL flag and drop the union, without masking or adding compound flags.
I can switch to that if you promise not to change your mind again. ;-)
Quoted 11 lines
Quoted 10 lines
but I'm not sure it would be less confusing.
Regarding isolating the sources of confusion in the framework rather than spreading them through drivers, I can't disagree, but I think that, for raw camera sensors at least, the best option would be to create a new camera sensor object with a much more tailored API than
v4l2_subdev(and of course wrapping that new object in av4l2_subdevin the framework). This won't address the other types of drivers, but I'm not sure we'll get any in the foreseable future.
Signature · 3 lines
Terveisin, Sakari Ailus
Laurent Pinchart
maintainerRe: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs
Hi Sakari,
Quoted 70 lines · Sakari Ailus
Quoted 60 lines · Laurent Pinchart
Quoted 59 lines · Laurent Pinchart
Quoted 48 lines · Sakari Ailus
Quoted 6 lines · Tomi Valkeinen
Quoted 3 lines · Sakari Ailus
Take the new
INTERNAL_SOURCEpad flag into account in validating routing IOCTL argument. Effectively this is a SINK pad in this respect. Due to the union there's no need to check for the particular name.What union? The one you add in the next patch?
Oops. I think we can reorder the patches.
Quoted 6 lines
As a concept the internal source pads sound good, and they work in practice in my tests. But this union is what grates me a bit. We have a flag,
MEDIA_PAD_FL_INTERNAL_SOURCE, which tells which field of the union to use, and then we go and do not use the new union field. Well, and also the naming, as we normally have source and sink pads, but now we also have internal source pads, which are actually identical to sink pads...The union still should be used by the user space. We're testing flags here and those flags are the same independently of the
INTERNAL_SOURCEflag.I'm fine by not adding that union though, but for the user space I think it's better we have it: explaining that the
sink_padhas a different meaning if that flag is set is harder than making it a union member.Quoted 9 lines
I understand the idea and reasoning, but the two points above do confuse me and I'm sure would confuse others also.
I wonder if it would be less or more confusing to simplify this by just adding a
MEDIA_PAD_FL_INTERNAL, which could be applied to either a source or a sink pad, and would prevent linking to it. The flag would indicate that the stream from/to that pad is generated/consumed internally. (I don't know when you would need an internal pad to consume data, but... who knows, the need might pop up =).This is another option. But I envision there will be more compatibility issues. Although... generally using such hardware requires knowledge of the device, and we haven't obviously had any support for devices needing this functionality in the tree. So in the end it might not matter much.
Quoted 8 lines
That would mean that an "internal sink pad" would generate a data stream, i.e. it's a "source", but I think a normal sink pad is almost the same
anyway: when considering entity's internal routing, the normal sink pad is a "source", and the same logic would apply with the internal pads too.An internal sink pad that generates a stream is a bit more confusing than an internal source pad, but I think that confusion is less than the rest of the confusion in the code-side that comes with the internal source pads.
Quoted 5 lines
I prefer having the possible sources of the confusion in the framework than in the drivers. Therefore I think
INTERNAL_SOURCEflag is a (slightly) better option.I wonder what Llaurent thinks.
I like the idea of a
MEDIA_PAD_FL_INTERNALflag. That's actually how I read patch 1/7, I didn't notice it used MEDIA_PAD_FL_INTERNAL*_SOURCE* :-)Quoted 8 lines
Quoted 3 lines
We could define
MEDIA_PAD_FL_INTERNAL_SOURCE#define
MEDIA_PAD_FL_INTERNAL_SOURCE(MEDIA_PAD_FL_SINK|MEDIA_PAD_FL_INTERNAL)One option would be to find terms different from sink and source in this case. It would generate less confusion to map (e.g., not a really good name)
MEDIA_PAD_FL_INTERNAL_PRODUCERtoMEDIA_PAD_FL_SINKand to thesink_padfield than usingMEDIA_PAD_FL_INTERNAL_SOURCE.
Quoted 7 lines
I don't think this helps as you'd still be accessing the sink pad related fields that are there for another purpose.
Alternatively I'd have the (plain) INTERNAL flag and drop the union, without masking or adding compound flags.
I can switch to that if you promise not to change your mind again. ;-)
I'll do my best :-) Could you show the impact (if any) it would have on drivers when accessing the routing fields ?
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
but I'm not sure it would be less confusing.
Regarding isolating the sources of confusion in the framework rather than spreading them through drivers, I can't disagree, but I think that, for raw camera sensors at least, the best option would be to create a new camera sensor object with a much more tailored API than
v4l2_subdev(and of course wrapping that new object in av4l2_subdevin the framework). This won't address the other types of drivers, but I'm not sure we'll get any in the foreseable future.
Signature · 3 lines
Regards, Laurent Pinchart
Sakari Ailus
maintainerRe: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs
Hi Laurent,
Quoted 76 lines · Laurent Pinchart
Hi Sakari,
Quoted 73 lines · Sakari Ailus
Quoted 60 lines · Laurent Pinchart
Quoted 59 lines · Laurent Pinchart
Quoted 48 lines · Sakari Ailus
Quoted 6 lines · Tomi Valkeinen
Quoted 3 lines · Sakari Ailus
Take the new
INTERNAL_SOURCEpad flag into account in validating routing IOCTL argument. Effectively this is a SINK pad in this respect. Due to the union there's no need to check for the particular name.What union? The one you add in the next patch?
Oops. I think we can reorder the patches.
Quoted 6 lines
As a concept the internal source pads sound good, and they work in practice in my tests. But this union is what grates me a bit. We have a flag,
MEDIA_PAD_FL_INTERNAL_SOURCE, which tells which field of the union to use, and then we go and do not use the new union field. Well, and also the naming, as we normally have source and sink pads, but now we also have internal source pads, which are actually identical to sink pads...The union still should be used by the user space. We're testing flags here and those flags are the same independently of the
INTERNAL_SOURCEflag.I'm fine by not adding that union though, but for the user space I think it's better we have it: explaining that the
sink_padhas a different meaning if that flag is set is harder than making it a union member.Quoted 9 lines
I understand the idea and reasoning, but the two points above do confuse me and I'm sure would confuse others also.
I wonder if it would be less or more confusing to simplify this by just adding a
MEDIA_PAD_FL_INTERNAL, which could be applied to either a source or a sink pad, and would prevent linking to it. The flag would indicate that the stream from/to that pad is generated/consumed internally. (I don't know when you would need an internal pad to consume data, but... who knows, the need might pop up =).This is another option. But I envision there will be more compatibility issues. Although... generally using such hardware requires knowledge of the device, and we haven't obviously had any support for devices needing this functionality in the tree. So in the end it might not matter much.
Quoted 8 lines
That would mean that an "internal sink pad" would generate a data stream, i.e. it's a "source", but I think a normal sink pad is almost the same
anyway: when considering entity's internal routing, the normal sink pad is a "source", and the same logic would apply with the internal pads too.An internal sink pad that generates a stream is a bit more confusing than an internal source pad, but I think that confusion is less than the rest of the confusion in the code-side that comes with the internal source pads.
Quoted 5 lines
I prefer having the possible sources of the confusion in the framework than in the drivers. Therefore I think
INTERNAL_SOURCEflag is a (slightly) better option.I wonder what Llaurent thinks.
I like the idea of a
MEDIA_PAD_FL_INTERNALflag. That's actually how I read patch 1/7, I didn't notice it used MEDIA_PAD_FL_INTERNAL*_SOURCE* :-)Quoted 8 lines
Quoted 3 lines
We could define
MEDIA_PAD_FL_INTERNAL_SOURCE#define
MEDIA_PAD_FL_INTERNAL_SOURCE(MEDIA_PAD_FL_SINK|MEDIA_PAD_FL_INTERNAL)One option would be to find terms different from sink and source in this case. It would generate less confusion to map (e.g., not a really good name)
MEDIA_PAD_FL_INTERNAL_PRODUCERtoMEDIA_PAD_FL_SINKand to thesink_padfield than usingMEDIA_PAD_FL_INTERNAL_SOURCE.I don't think this helps as you'd still be accessing the sink pad related fields that are there for another purpose.
Quoted 7 lines
Quoted 4 lines
Alternatively I'd have the (plain) INTERNAL flag and drop the union, without masking or adding compound flags.
I can switch to that if you promise not to change your mind again. ;-)
I'll do my best :-) Could you show the impact (if any) it would have on drivers when accessing the routing fields ?
I don't think there's much of an impact for the drivers. It's mainly affecting setting up pads for the entities. Tomi?
Signature · 1 line
Sakari Ailus
Laurent Pinchart
maintainerRe: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs
Quoted 78 lines · Sakari Ailus
Quoted 77 lines · Laurent Pinchart
Quoted 76 lines · Sakari Ailus
Quoted 60 lines · Laurent Pinchart
Quoted 59 lines · Laurent Pinchart
Quoted 48 lines · Sakari Ailus
Quoted 6 lines · Tomi Valkeinen
Quoted 3 lines · Sakari Ailus
Take the new
INTERNAL_SOURCEpad flag into account in validating routing IOCTL argument. Effectively this is a SINK pad in this respect. Due to the union there's no need to check for the particular name.What union? The one you add in the next patch?
Oops. I think we can reorder the patches.
Quoted 6 lines
As a concept the internal source pads sound good, and they work in practice in my tests. But this union is what grates me a bit. We have a flag,
MEDIA_PAD_FL_INTERNAL_SOURCE, which tells which field of the union to use, and then we go and do not use the new union field. Well, and also the naming, as we normally have source and sink pads, but now we also have internal source pads, which are actually identical to sink pads...The union still should be used by the user space. We're testing flags here and those flags are the same independently of the
INTERNAL_SOURCEflag.I'm fine by not adding that union though, but for the user space I think it's better we have it: explaining that the
sink_padhas a different meaning if that flag is set is harder than making it a union member.Quoted 9 lines
I understand the idea and reasoning, but the two points above do confuse me and I'm sure would confuse others also.
I wonder if it would be less or more confusing to simplify this by just adding a
MEDIA_PAD_FL_INTERNAL, which could be applied to either a source or a sink pad, and would prevent linking to it. The flag would indicate that the stream from/to that pad is generated/consumed internally. (I don't know when you would need an internal pad to consume data, but... who knows, the need might pop up =).This is another option. But I envision there will be more compatibility issues. Although... generally using such hardware requires knowledge of the device, and we haven't obviously had any support for devices needing this functionality in the tree. So in the end it might not matter much.
Quoted 8 lines
That would mean that an "internal sink pad" would generate a data stream, i.e. it's a "source", but I think a normal sink pad is almost the same
anyway: when considering entity's internal routing, the normal sink pad is a "source", and the same logic would apply with the internal pads too.An internal sink pad that generates a stream is a bit more confusing than an internal source pad, but I think that confusion is less than the rest of the confusion in the code-side that comes with the internal source pads.
Quoted 5 lines
I prefer having the possible sources of the confusion in the framework than in the drivers. Therefore I think
INTERNAL_SOURCEflag is a (slightly) better option.I wonder what Llaurent thinks.
I like the idea of a
MEDIA_PAD_FL_INTERNALflag. That's actually how I read patch 1/7, I didn't notice it used MEDIA_PAD_FL_INTERNAL*_SOURCE* :-)Quoted 8 lines
Quoted 3 lines
We could define
MEDIA_PAD_FL_INTERNAL_SOURCE#define
MEDIA_PAD_FL_INTERNAL_SOURCE(MEDIA_PAD_FL_SINK|MEDIA_PAD_FL_INTERNAL)One option would be to find terms different from sink and source in this case. It would generate less confusion to map (e.g., not a really good name)
MEDIA_PAD_FL_INTERNAL_PRODUCERtoMEDIA_PAD_FL_SINKand to thesink_padfield than usingMEDIA_PAD_FL_INTERNAL_SOURCE.I don't think this helps as you'd still be accessing the sink pad related fields that are there for another purpose.
Alternatively I'd have the (plain) INTERNAL flag and drop the union, without masking or adding compound flags.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
I can switch to that if you promise not to change your mind again. ;-)
I'll do my best :-) Could you show the impact (if any) it would have on drivers when accessing the routing fields ?
I don't think there's much of an impact for the drivers. It's mainly affecting setting up pads for the entities. Tomi?
I think it would be useful to discuss it with an actual sensor driver using the API.
Signature · 3 lines
Regards, Laurent Pinchart
Re: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs
In reply to Sakari AilusQuoted 24 lines · Sakari Ailus
Take the new
INTERNAL_SOURCEpad flag into account in validating routing IOCTL argument. Effectively this is a SINK pad in this respect. Due to the union there's no need to check for the particular name.Signed-off-by Sakari Ailus <[email protected]>--- drivers/media/v4l2-core/v4l2-subdev.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)drivers/media/v4l2-core/v4l2-subdev.c +2
-1@@ -906,7 +906,8 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,return -EINVAL ;if (!(pads[route->sink_pad].flags & -MEDIA_PAD_FL_SINK )) + (MEDIA_PAD_FL_SINK | +MEDIA_PAD_FL_INTERNAL_SOURCE )))return -EINVAL ;if (route->source_pad >= sd->entity.num_pads)
drivers/media/v4l2-core/v4l2-subdev.c int v4l2_subdev_routing_validate(struct v4l2_subdev *sd, · line 1792
Quoted 6 lines
drivers/media/v4l2-core/v4l2-subdev.c +2
-1@@ -1787,7 +1788,8 @@ int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,/* Validate the sink and source pad numbers. */ if (route->sink_pad >= sd->entity.num_pads || - !(sd->entity.pads[route->sink_pad].flags &MEDIA_PAD_FL_SINK )) { + !(sd->entity.pads[route->sink_pad].flags & + (MEDIA_PAD_FL_SINK |MEDIA_PAD_FL_INTERNAL_SOURCE ))) {
So this relates to my reply to the previous patch: calling a sink pad an INTERNAL_SOURCE is really confusing.
Some suggestions for alternative names:
1) MEDIA_PAD_FL_SINK_INTERNAL
2) MEDIA_PAD_FL_SINK_INTERNAL_SOURCE (or _SRC)
3) MEDIA_PAD_FL_SINK_FROM_INTERNAL_SOURCE (or _SRC)
IMHO the name *has* to start with MEDIA_PAD_FL_SINK_ to indicate that it is a sink pad, followed by something that indicates that it is fed from an internal source.
Regards,
Hans
Quoted 3 lines
drivers/media/v4l2-core/v4l2-subdev.c +0
-0dev_dbg (sd->dev,"route %u sink (%u) is not a sink pad\n" , i, route->sink_pad); goto out;
Sakari Ailus
maintainerRFC 3/7 media: uapi: v4l: Document source routes
In reply to Sakari AilusDocument how internal pads are used on source routes.
--- .../userspace-api/media/v4l/dev-subdev.rst | 18 ++++++++++++++++++ .../media/v4l/vidioc-subdev-g-routing.rst | 5 +++++ include/uapi/linux/v4l2-subdev.h | 6 +++++- 3 files changed, 28 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/v4l/dev-subdev.rst +18 -0
@@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the sub-device and a (pad, stream) pair. For sub-devices that do not support multiplexed streams the 'stream' field is always 0. +.. _v4l2-subdev-source-routes: + +Source routes +^^^^^^^^^^^^^ + +Cases where a single sub-device pad is a source of multiple streams are special +as there is no sink pad for such a route. In those cases, an internal pad is +used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCE +<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set. + +Internal pads have all the properties of a sink pad in such case, including +formats and selections. The format in this case is the source format of the +stream. An internal pad always has a single stream only (0). + +Generally source routes are not modifiable but they can be activated and +inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE +<v4l2-subdev-routing-flags>` flag, depending on driver capabilities. + Interaction between routes, streams, formats and selections ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst +5 -0
@@ -94,6 +94,11 @@ for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again. * - __u32 - ``sink_pad`` - Sink pad number. + * - __u32 + - ``internal_source_pad`` + - Internal source pad number. For pads with :ref:`internal source pad flag + <MEDIA-PAD-FL-INTERNAL-SOURCE>`, for use with :ref:`source routes + <v4l2-subdev-source-routes>`. * - __u32 - ``sink_stream`` - Sink pad stream number.
include/uapi/linux/v4l2-subdev.h +5 -1
@@ -203,6 +203,7 @@ struct v4l2_subdev_capability {* struct v4l2_subdev_route - A route inside a subdev * * @sink_pad: the sink pad index +* @internal_source_pad: the internal source pad * @sink_stream: the sink stream identifier * @source_pad: the source pad index * @source_stream: the source stream identifier
@@ -210,7 +211,10 @@ struct v4l2_subdev_capability {* @reserved: drivers and applications must zero this array */ struct v4l2_subdev_route { -__u32 sink_pad; +union { +__u32 sink_pad; +__u32 internal_source_pad; + };__u32 sink_stream;__u32 source_pad;__u32 source_stream;
Signature · 1 line
2.30.2
Re: [RFC 3/7] media: uapi: v4l: Document source routes
Quoted 25 lines · Sakari Ailus
Document how internal pads are used on source routes.
Signed-off-by Sakari Ailus <[email protected]>--- .../userspace-api/media/v4l/dev-subdev.rst | 18 ++++++++++++++++++ .../media/v4l/vidioc-subdev-g-routing.rst | 5 +++++ include/uapi/linux/v4l2-subdev.h | 6 +++++- 3 files changed, 28 insertions(+), 1 deletion(-)Documentation/userspace-api/media/v4l/dev-subdev.rst +8
-0@@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the sub-device and a (pad, stream) pair. For sub-devices that do not support multiplexed streams the 'stream' field is always 0. +.. _v4l2-subdev-source-routes: + +Source routes +^^^^^^^^^^^^^ + +Cases where a single sub-device pad is a source of multiple streams are special +as there is no sink pad for such a route. In those cases, an internal pad is +used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCE
Documentation/userspace-api/media/v4l/dev-subdev.rst A stream at a specific point in the media pipeline is identified by the · line 571
Quoted 10 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +10
-0+<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set. + +Internal pads have all the properties of a sink pad in such case, including +formats and selections. The format in this case is the source format of the +stream. An internal pad always has a single stream only (0). + +Generally source routes are not modifiable but they can be activated and +inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE +<v4l2-subdev-routing-flags>` flag, depending on driver capabilities. +
I find the above chapter a bit difficult to read, but I think we need to conclude on the internal-pad vs internal-source-pad discussion first. However, one point/question:
You write that there's only one stream in an internal pad. I wonder if that's a good or a necessary limitation... Do you see that allowing multiple streams brings extra complexity? It's still up to each driver to decide how many streams they support (most would just support a single one), so each driver can easily enforce that limit.
I'm fine with starting with only single-stream support, but if we later need to loosen that restriction, I wonder if it'll be difficult if we have specified that there can be only one stream. I.e. will the drivers and the userspace depend on that limit.
Tomi
Sakari Ailus
maintainerRe: [RFC 3/7] media: uapi: v4l: Document source routes
Moi,
Quoted 40 lines · Tomi Valkeinen
Quoted 24 lines · Sakari Ailus
Document how internal pads are used on source routes.
Signed-off-by Sakari Ailus <[email protected]>--- .../userspace-api/media/v4l/dev-subdev.rst | 18 ++++++++++++++++++ .../media/v4l/vidioc-subdev-g-routing.rst | 5 +++++ include/uapi/linux/v4l2-subdev.h | 6 +++++- 3 files changed, 28 insertions(+), 1 deletion(-)Documentation/userspace-api/media/v4l/dev-subdev.rst +8
-0@@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the sub-device and a (pad, stream) pair. For sub-devices that do not support multiplexed streams the 'stream' field is always 0. +.. _v4l2-subdev-source-routes: + +Source routes +^^^^^^^^^^^^^ + +Cases where a single sub-device pad is a source of multiple streams are special +as there is no sink pad for such a route. In those cases, an internal pad is +used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCEDocumentation/userspace-api/media/v4l/dev-subdev.rst A stream at a specific point in the media pipeline is identified by the · line 570
Quoted 10 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +10
-0+<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set. + +Internal pads have all the properties of a sink pad in such case, including +formats and selections. The format in this case is the source format of the +stream. An internal pad always has a single stream only (0). + +Generally source routes are not modifiable but they can be activated and +inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE +<v4l2-subdev-routing-flags>` flag, depending on driver capabilities. +I find the above chapter a bit difficult to read, but I think we need to conclude on the internal-pad vs internal-source-pad discussion first. However, one point/question:
Quoted 5 lines
You write that there's only one stream in an internal pad. I wonder if that's a good or a necessary limitation... Do you see that allowing multiple streams brings extra complexity? It's still up to each driver to decide how many streams they support (most would just support a single one), so each driver can easily enforce that limit.
Good question. As we don't seem to have a tangible reason to allow it, I'd deny it until there's a use case.
Or did you have a use case in mind?
I thought indicating which streams will be bound together (i.e. either are all disabled or enabled) could be one, but I'm not sure we need that at the moment at least.
Quoted 5 lines
I'm fine with starting with only single-stream support, but if we later need to loosen that restriction, I wonder if it'll be difficult if we have specified that there can be only one stream. I.e. will the drivers and the userspace depend on that limit.
We can always allow what wasn't allowed previously so that's a non-issue, really. But it needs to bring new functionality, otherwise there's no reason to do that.
Signature · 3 lines
Kind regards, Sakari Ailus
Re: [RFC 3/7] media: uapi: v4l: Document source routes
Quoted 49 lines · Sakari Ailus
Moi,
Quoted 40 lines · Tomi Valkeinen
Quoted 24 lines · Sakari Ailus
Document how internal pads are used on source routes.
Signed-off-by Sakari Ailus <[email protected]>--- .../userspace-api/media/v4l/dev-subdev.rst | 18 ++++++++++++++++++ .../media/v4l/vidioc-subdev-g-routing.rst | 5 +++++ include/uapi/linux/v4l2-subdev.h | 6 +++++- 3 files changed, 28 insertions(+), 1 deletion(-)Documentation/userspace-api/media/v4l/dev-subdev.rst +8
-0@@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the sub-device and a (pad, stream) pair. For sub-devices that do not support multiplexed streams the 'stream' field is always 0. +.. _v4l2-subdev-source-routes: + +Source routes +^^^^^^^^^^^^^ + +Cases where a single sub-device pad is a source of multiple streams are special +as there is no sink pad for such a route. In those cases, an internal pad is +used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCEDocumentation/userspace-api/media/v4l/dev-subdev.rst A stream at a specific point in the media pipeline is identified by the · line 570
Quoted 10 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +10
-0+<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set. + +Internal pads have all the properties of a sink pad in such case, including +formats and selections. The format in this case is the source format of the +stream. An internal pad always has a single stream only (0). + +Generally source routes are not modifiable but they can be activated and +inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE +<v4l2-subdev-routing-flags>` flag, depending on driver capabilities. +I find the above chapter a bit difficult to read, but I think we need to conclude on the internal-pad vs internal-source-pad discussion first. However, one point/question:
Quoted 5 lines
You write that there's only one stream in an internal pad. I wonder if that's a good or a necessary limitation... Do you see that allowing multiple streams brings extra complexity? It's still up to each driver to decide how many streams they support (most would just support a single one), so each driver can easily enforce that limit.
Quoted 8 lines
Good question. As we don't seem to have a tangible reason to allow it, I'd deny it until there's a use case.
Or did you have a use case in mind?
I thought indicating which streams will be bound together (i.e. either are all disabled or enabled) could be one, but I'm not sure we need that at the moment at least.
I don't have nothing concrete in mind. Maybe a TPG which also generates some kind of metadata. But that could be implemented as two internal pads.
Quoted 8 lines
Quoted 4 lines
I'm fine with starting with only single-stream support, but if we later need to loosen that restriction, I wonder if it'll be difficult if we have specified that there can be only one stream. I.e. will the drivers and the userspace depend on that limit.
We can always allow what wasn't allowed previously so that's a non-issue, really. But it needs to bring new functionality, otherwise there's no reason to do that.
It's not always that easy. If the drivers and the userspace expect that there's a single route with ID 0, and then with a new kernel there are more streams or the single stream is ID 1, things could go wrong.
Tomi
Sakari Ailus
maintainerRe: [RFC 3/7] media: uapi: v4l: Document source routes
Moi,
Quoted 67 lines · Tomi Valkeinen
Quoted 49 lines · Sakari Ailus
Moi,
Quoted 40 lines · Tomi Valkeinen
Quoted 24 lines · Sakari Ailus
Document how internal pads are used on source routes.
Signed-off-by Sakari Ailus <[email protected]>--- .../userspace-api/media/v4l/dev-subdev.rst | 18 ++++++++++++++++++ .../media/v4l/vidioc-subdev-g-routing.rst | 5 +++++ include/uapi/linux/v4l2-subdev.h | 6 +++++- 3 files changed, 28 insertions(+), 1 deletion(-)Documentation/userspace-api/media/v4l/dev-subdev.rst +8
-0@@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the sub-device and a (pad, stream) pair. For sub-devices that do not support multiplexed streams the 'stream' field is always 0. +.. _v4l2-subdev-source-routes: + +Source routes +^^^^^^^^^^^^^ + +Cases where a single sub-device pad is a source of multiple streams are special +as there is no sink pad for such a route. In those cases, an internal pad is +used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCEDocumentation/userspace-api/media/v4l/dev-subdev.rst A stream at a specific point in the media pipeline is identified by the · line 570
Quoted 10 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +10
-0+<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set. + +Internal pads have all the properties of a sink pad in such case, including +formats and selections. The format in this case is the source format of the +stream. An internal pad always has a single stream only (0). + +Generally source routes are not modifiable but they can be activated and +inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE +<v4l2-subdev-routing-flags>` flag, depending on driver capabilities. +I find the above chapter a bit difficult to read, but I think we need to conclude on the internal-pad vs internal-source-pad discussion first. However, one point/question:
Quoted 5 lines
You write that there's only one stream in an internal pad. I wonder if that's a good or a necessary limitation... Do you see that allowing multiple streams brings extra complexity? It's still up to each driver to decide how many streams they support (most would just support a single one), so each driver can easily enforce that limit.
Quoted 8 lines
Good question. As we don't seem to have a tangible reason to allow it, I'd deny it until there's a use case.
Or did you have a use case in mind?
I thought indicating which streams will be bound together (i.e. either are all disabled or enabled) could be one, but I'm not sure we need that at the moment at least.
I don't have nothing concrete in mind. Maybe a TPG which also generates some kind of metadata. But that could be implemented as two internal pads.
Quoted 5 lines
Quoted 4 lines
I'm fine with starting with only single-stream support, but if we later need to loosen that restriction, I wonder if it'll be difficult if we have specified that there can be only one stream. I.e. will the drivers and the userspace depend on that limit.
Quoted 7 lines
Quoted 3 lines
We can always allow what wasn't allowed previously so that's a non-issue, really. But it needs to bring new functionality, otherwise there's no reason to do that.
It's not always that easy. If the drivers and the userspace expect that there's a single route with ID 0, and then with a new kernel there are more streams or the single stream is ID 1, things could go wrong.
Other drivers are a non-issue IMO as this isn't visible to other drivers.
I think for user space this could be an issue if it were entirely generic. But that's probably not going to be the case, is it? I don't mean test programs.
Signature · 3 lines
Terveisin, Sakari Ailus
Laurent Pinchart
maintainerRe: [RFC 3/7] media: uapi: v4l: Document source routes
Quoted 74 lines · Sakari Ailus
Quoted 65 lines · Tomi Valkeinen
Quoted 47 lines · Sakari Ailus
Quoted 40 lines · Tomi Valkeinen
Quoted 24 lines · Sakari Ailus
Document how internal pads are used on source routes.
Signed-off-by Sakari Ailus <[email protected]>--- .../userspace-api/media/v4l/dev-subdev.rst | 18 ++++++++++++++++++ .../media/v4l/vidioc-subdev-g-routing.rst | 5 +++++ include/uapi/linux/v4l2-subdev.h | 6 +++++- 3 files changed, 28 insertions(+), 1 deletion(-)Documentation/userspace-api/media/v4l/dev-subdev.rst +8
-0@@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the sub-device and a (pad, stream) pair. For sub-devices that do not support multiplexed streams the 'stream' field is always 0. +.. _v4l2-subdev-source-routes: + +Source routes +^^^^^^^^^^^^^ + +Cases where a single sub-device pad is a source of multiple streams are special +as there is no sink pad for such a route. In those cases, an internal pad is +used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCEDocumentation/userspace-api/media/v4l/dev-subdev.rst A stream at a specific point in the media pipeline is identified by the · line 570
Quoted 10 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +10
-0+<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set. + +Internal pads have all the properties of a sink pad in such case, including +formats and selections. The format in this case is the source format of the +stream. An internal pad always has a single stream only (0). + +Generally source routes are not modifiable but they can be activated and +inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE +<v4l2-subdev-routing-flags>` flag, depending on driver capabilities. +I find the above chapter a bit difficult to read, but I think we need to conclude on the internal-pad vs internal-source-pad discussion first. However, one point/question:
Quoted 5 lines
You write that there's only one stream in an internal pad. I wonder if that's a good or a necessary limitation... Do you see that allowing multiple streams brings extra complexity? It's still up to each driver to decide how many streams they support (most would just support a single one), so each driver can easily enforce that limit.
Quoted 8 lines
Good question. As we don't seem to have a tangible reason to allow it, I'd deny it until there's a use case.
Or did you have a use case in mind?
I thought indicating which streams will be bound together (i.e. either are all disabled or enabled) could be one, but I'm not sure we need that at the moment at least.
I don't have nothing concrete in mind. Maybe a TPG which also generates some kind of metadata. But that could be implemented as two internal pads.
Quoted 5 lines
Quoted 4 lines
I'm fine with starting with only single-stream support, but if we later need to loosen that restriction, I wonder if it'll be difficult if we have specified that there can be only one stream. I.e. will the drivers and the userspace depend on that limit.
Quoted 7 lines
Quoted 3 lines
We can always allow what wasn't allowed previously so that's a non-issue, really. But it needs to bring new functionality, otherwise there's no reason to do that.
It's not always that easy. If the drivers and the userspace expect that there's a single route with ID 0, and then with a new kernel there are more streams or the single stream is ID 1, things could go wrong.
Quoted 5 lines
Other drivers are a non-issue IMO as this isn't visible to other drivers.
I think for user space this could be an issue if it were entirely generic. But that's probably not going to be the case, is it? I don't mean test programs.
I could imagine this breaking libcamera for instance.
Signature · 3 lines
Regards, Laurent Pinchart
Laurent Pinchart
maintainerRe: [RFC 3/7] media: uapi: v4l: Document source routes
In reply to Tomi ValkeinenHello,
Quoted 33 lines · Tomi Valkeinen
Quoted 32 lines · Sakari Ailus
Quoted 31 lines · Tomi Valkeinen
Quoted 30 lines · Sakari Ailus
Document how internal pads are used on source routes.
Signed-off-by Sakari Ailus <[email protected]>--- .../userspace-api/media/v4l/dev-subdev.rst | 18 ++++++++++++++++++ .../media/v4l/vidioc-subdev-g-routing.rst | 5 +++++ include/uapi/linux/v4l2-subdev.h | 6 +++++- 3 files changed, 28 insertions(+), 1 deletion(-)Documentation/userspace-api/media/v4l/dev-subdev.rst +14
-0@@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the sub-device and a (pad, stream) pair. For sub-devices that do not support multiplexed streams the 'stream' field is always 0. +.. _v4l2-subdev-source-routes: + +Source routes +^^^^^^^^^^^^^ + +Cases where a single sub-device pad is a source of multiple streams are special +as there is no sink pad for such a route. In those cases, an internal pad is +used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCE +<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set. + +Internal pads have all the properties of a sink pad in such case, including +formats and selections. The format in this case is the source format of the +stream. An internal pad always has a single stream only (0). +
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst A stream at a specific point in the media pipeline is identified by the · line 571
Quoted 4 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +4
-0+Generally source routes are not modifiable but they can be activated and +inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE +<v4l2-subdev-routing-flags>` flag, depending on driver capabilities. +I find the above chapter a bit difficult to read, but I think we need to conclude on the internal-pad vs internal-source-pad discussion first. However, one point/question:
Agreed, it's far from being clear :-( Even the first sentence, "Cases where a single sub-device pad is a source of multiple streams are special" is confusing, having a subdev source pad carrying multiple streams isn't special, it is found in, for instance, FPD-Link or GMSL receivers that transmit multiple streams from different cameras over a single CSI-2 output. This may not be what Sakari meant here, but it can be understood that way.
We need more than 3 paragraphs here, and we need a very clear example with a diagram. I'd recommend using a camera sensor that produces image data and embedded data for the example, as that's a common real-life use case. The text should present the example, explain what the problem is, and then explain how internal pads fix it. Then it can expand on the features and usage of internal pads in a more generic way.
Quoted 23 lines
Quoted 6 lines
Quoted 5 lines
You write that there's only one stream in an internal pad. I wonder if that's a good or a necessary limitation... Do you see that allowing multiple streams brings extra complexity? It's still up to each driver to decide how many streams they support (most would just support a single one), so each driver can easily enforce that limit.
Quoted 8 lines
Good question. As we don't seem to have a tangible reason to allow it, I'd deny it until there's a use case.
Or did you have a use case in mind?
I thought indicating which streams will be bound together (i.e. either are all disabled or enabled) could be one, but I'm not sure we need that at the moment at least.
I don't have nothing concrete in mind. Maybe a TPG which also generates some kind of metadata. But that could be implemented as two internal pads.
Quoted 5 lines
Quoted 4 lines
I'm fine with starting with only single-stream support, but if we later need to loosen that restriction, I wonder if it'll be difficult if we have specified that there can be only one stream. I.e. will the drivers and the userspace depend on that limit.
Quoted 7 lines
Quoted 3 lines
We can always allow what wasn't allowed previously so that's a non-issue, really. But it needs to bring new functionality, otherwise there's no reason to do that.
It's not always that easy. If the drivers and the userspace expect that there's a single route with ID 0, and then with a new kernel there are more streams or the single stream is ID 1, things could go wrong.
I agree with Tomi here. On the kernel side it should be fine (with an unknown amount of pain), but I'd consider this as a userspace API breakage. If the specifications indicates that only a single stream can be used, applications may rely on that, and things could go wrong if new streams are added.
We can restrict the kernel implementation to a single stream, but the userspace API has to indicate that multiple streams can co-exist if we want to allow that in the future.
Signature · 3 lines
Regards, Laurent Pinchart
Sakari Ailus
maintainerRe: [RFC 3/7] media: uapi: v4l: Document source routes
Hi Laurent,
Quoted 45 lines · Laurent Pinchart
Hello,
Quoted 33 lines · Tomi Valkeinen
Quoted 32 lines · Sakari Ailus
Quoted 31 lines · Tomi Valkeinen
Quoted 30 lines · Sakari Ailus
Document how internal pads are used on source routes.
Signed-off-by Sakari Ailus <[email protected]>--- .../userspace-api/media/v4l/dev-subdev.rst | 18 ++++++++++++++++++ .../media/v4l/vidioc-subdev-g-routing.rst | 5 +++++ include/uapi/linux/v4l2-subdev.h | 6 +++++- 3 files changed, 28 insertions(+), 1 deletion(-)Documentation/userspace-api/media/v4l/dev-subdev.rst +14
-0@@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the sub-device and a (pad, stream) pair. For sub-devices that do not support multiplexed streams the 'stream' field is always 0. +.. _v4l2-subdev-source-routes: + +Source routes +^^^^^^^^^^^^^ + +Cases where a single sub-device pad is a source of multiple streams are special +as there is no sink pad for such a route. In those cases, an internal pad is +used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCE +<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set. + +Internal pads have all the properties of a sink pad in such case, including +formats and selections. The format in this case is the source format of the +stream. An internal pad always has a single stream only (0). +Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst A stream at a specific point in the media pipeline is identified by the · line 571
Quoted 4 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +4
-0+Generally source routes are not modifiable but they can be activated and +inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE +<v4l2-subdev-routing-flags>` flag, depending on driver capabilities. +I find the above chapter a bit difficult to read, but I think we need to conclude on the internal-pad vs internal-source-pad discussion first. However, one point/question:
Quoted 7 lines
Agreed, it's far from being clear :-( Even the first sentence, "Cases where a single sub-device pad is a source of multiple streams are special" is confusing, having a subdev source pad carrying multiple streams isn't special, it is found in, for instance, FPD-Link or GMSL receivers that transmit multiple streams from different cameras over a single CSI-2 output. This may not be what Sakari meant here, but it can be understood that way.
The sentence continues on the next line. I'll rework this part of the documentation in any case while moving to a plain INTERNAL flag.
Quoted 7 lines
We need more than 3 paragraphs here, and we need a very clear example with a diagram. I'd recommend using a camera sensor that produces image data and embedded data for the example, as that's a common real-life use case. The text should present the example, explain what the problem is, and then explain how internal pads fix it. Then it can expand on the features and usage of internal pads in a more generic way.
It's fine to have examples but I'd add them to where the other examples are located.
Quoted 38 lines
Quoted 24 lines
Quoted 7 lines
Quoted 6 lines
You write that there's only one stream in an internal pad. I wonder if that's a good or a necessary limitation... Do you see that allowing multiple streams brings extra complexity? It's still up to each driver to decide how many streams they support (most would just support a single one), so each driver can easily enforce that limit.
Quoted 8 lines
Good question. As we don't seem to have a tangible reason to allow it, I'd deny it until there's a use case.
Or did you have a use case in mind?
I thought indicating which streams will be bound together (i.e. either are all disabled or enabled) could be one, but I'm not sure we need that at the moment at least.
I don't have nothing concrete in mind. Maybe a TPG which also generates some kind of metadata. But that could be implemented as two internal pads.
Quoted 5 lines
Quoted 4 lines
I'm fine with starting with only single-stream support, but if we later need to loosen that restriction, I wonder if it'll be difficult if we have specified that there can be only one stream. I.e. will the drivers and the userspace depend on that limit.
Quoted 7 lines
Quoted 3 lines
We can always allow what wasn't allowed previously so that's a non-issue, really. But it needs to bring new functionality, otherwise there's no reason to do that.
It's not always that easy. If the drivers and the userspace expect that there's a single route with ID 0, and then with a new kernel there are more streams or the single stream is ID 1, things could go wrong.
I agree with Tomi here. On the kernel side it should be fine (with an unknown amount of pain), but I'd consider this as a userspace API breakage. If the specifications indicates that only a single stream can be used, applications may rely on that, and things could go wrong if new streams are added.
Quoted 3 lines
We can restrict the kernel implementation to a single stream, but the userspace API has to indicate that multiple streams can co-exist if we want to allow that in the future.
I doubt this ever could be an actual problem. Allowing multiple streams to be merged on a source pad requires the user to first configure them.
I find it hard to believe there would be an application that tried to configure routes this way (which would have always failed) and then proceeded anyway, and somehow returning success from route setup leads to routing configuration that the application couldn't handle anymore.
Signature · 3 lines
Kind regards, Sakari Ailus
Re: [RFC 3/7] media: uapi: v4l: Document source routes
In reply to Sakari AilusQuoted 16 lines · Sakari Ailus
Document how internal pads are used on source routes.
Signed-off-by Sakari Ailus <[email protected]>--- .../userspace-api/media/v4l/dev-subdev.rst | 18 ++++++++++++++++++ .../media/v4l/vidioc-subdev-g-routing.rst | 5 +++++ include/uapi/linux/v4l2-subdev.h | 6 +++++- 3 files changed, 28 insertions(+), 1 deletion(-)Documentation/userspace-api/media/v4l/dev-subdev.rst +0
-0@@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the sub-device and a (pad, stream) pair. For sub-devices that do not support multiplexed streams the 'stream' field is always 0.
Documentation/userspace-api/media/v4l/dev-subdev.rst A stream at a specific point in the media pipeline is identified by the · line 562
Quoted 10 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +9
-0+.. _v4l2-subdev-source-routes: + +Source routes +^^^^^^^^^^^^^ + +Cases where a single sub-device pad is a source of multiple streams are special +as there is no sink pad for such a route. In those cases, an internal pad is +used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCE +<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set.
add: instead of the <MEDIA-PAD-FL-SINK> flag.
Documentation/userspace-api/media/v4l/dev-subdev.rst A stream at a specific point in the media pipeline is identified by the · line 569
Quoted 7 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +7
-0+ +Internal pads have all the properties of a sink pad in such case, including +formats and selections. The format in this case is the source format of the +stream. An internal pad always has a single stream only (0). + +Generally source routes are not modifiable but they can be activated and +inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
inactivated -> deactivated
Documentation/userspace-api/media/v4l/dev-subdev.rst A stream at a specific point in the media pipeline is identified by the · line 571
Quoted 2 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +2
-0+<v4l2-subdev-routing-flags>` flag, depending on driver capabilities. +
Again, this really needs one or more examples.
Quoted 32 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +0
-0Interaction between routes, streams, formats and selections ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst +5
-0@@ -94,6 +94,11 @@ for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again. * - __u32 - ``sink_pad`` - Sink pad number. + * - __u32 + - ``internal_source_pad`` + - Internal source pad number. For pads with :ref:`internal source pad flag + <MEDIA-PAD-FL-INTERNAL-SOURCE>`, for use with :ref:`source routes + <v4l2-subdev-source-routes>`. * - __u32 - ``sink_stream`` - Sink pad stream number.include/uapi/linux/v4l2-subdev.h +1
-0@@ -203,6 +203,7 @@ struct v4l2_subdev_capability {* struct v4l2_subdev_route - A route inside a subdev * * @sink_pad: the sink pad index +* @internal_source_pad: the internal source pad * @sink_stream: the sink stream identifier * @source_pad: the source pad index * @source_stream: the source stream identifier
include/uapi/linux/v4l2-subdev.h struct v4l2_subdev_capability { · line 216
Quoted 7 lines
include/uapi/linux/v4l2-subdev.h +3
-1@@ -210,7 +211,10 @@ struct v4l2_subdev_capability {* @reserved: drivers and applications must zero this array */ struct v4l2_subdev_route { -__u32 sink_pad; +union { +__u32 sink_pad; +__u32 internal_source_pad;
Nobody will understand this. I think this should still be kept as sink_pad. If the INTERNAL_SOURCE flag is changed to SINK_FROM_INTERNAL_SOURCE (or something along those lines), then 'sink_pad' still makes sense and you don't need this change.
It's still a sink pad, just a sink pad with a special property in that it receives the data from an internal source.
include/uapi/linux/v4l2-subdev.h struct v4l2_subdev_capability { · line 220
Quoted 4 lines
include/uapi/linux/v4l2-subdev.h +1
-0+ };__u32 sink_stream;__u32 source_pad;__u32 source_stream;
Regards,
Hans
Sakari Ailus
maintainerRFC 4/7 media: mc: Check pad flag validity
In reply to Sakari AilusCheck the validity of pad flags on entity init. Exactly one of the flags must be set.
--- drivers/media/mc/mc-entity.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
drivers/media/mc/mc-entity.c +11 -1
@@ -197,6 +197,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,struct media_device *mdev = entity->graph_obj.mdev;struct media_pad *iter;unsigned int i =0 ; +int ret =0 ;if (num_pads >=MEDIA_ENTITY_MAX_PADS )return -E2BIG ;
@@ -210,6 +211,15 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,media_entity_for_each_pad (entity, iter) { iter->entity = entity; iter->index = i++; + +if (hweight32 (iter->flags & (MEDIA_PAD_FL_SINK | +MEDIA_PAD_FL_SOURCE | +MEDIA_PAD_FL_INTERNAL_SOURCE )) + !=1 ) { + ret = -EINVAL ; +break ; + } +if (mdev)media_gobj_create (mdev,MEDIA_GRAPH_PAD , &iter->graph_obj);
@@ -218,7 +228,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,if (mdev)mutex_unlock (&mdev->graph_mutex); -return 0 ; +return ret; }EXPORT_SYMBOL_GPL (media_entity_pads_init);
Signature · 1 line
2.30.2
Laurent Pinchart
maintainerRe: [RFC 4/7] media: mc: Check pad flag validity
Hi Sakari,
Thank you for the patch.
Quoted 21 lines · Sakari Ailus
Check the validity of pad flags on entity init. Exactly one of the flags must be set.
Signed-off-by Sakari Ailus <[email protected]>--- drivers/media/mc/mc-entity.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)drivers/media/mc/mc-entity.c +1
-0@@ -197,6 +197,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,struct media_device *mdev = entity->graph_obj.mdev;struct media_pad *iter;unsigned int i =0 ; +int ret =0 ;if (num_pads >=MEDIA_ENTITY_MAX_PADS )return -E2BIG ;
drivers/media/mc/mc-entity.c int media_entity_pads_init(struct media_entity *entity, u16 num_pads, · line 218
Quoted 8 lines
drivers/media/mc/mc-entity.c +5
-0@@ -210,6 +211,15 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,media_entity_for_each_pad (entity, iter) { iter->entity = entity; iter->index = i++; + +if (hweight32 (iter->flags & (MEDIA_PAD_FL_SINK | +MEDIA_PAD_FL_SOURCE | +MEDIA_PAD_FL_INTERNAL_SOURCE )) + !=1 ) {
I would add this check as patch 1/7 in this series, with FL_SINK an FL_SOURCE only, and then extend it in the patch where you introduce FL_INTERNAL_SOURCE.
Quoted 16 lines
drivers/media/mc/mc-entity.c +5
-1+ ret = -EINVAL ; +break ; + } +if (mdev)media_gobj_create (mdev,MEDIA_GRAPH_PAD , &iter->graph_obj);@@ -218,7 +228,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,if (mdev)mutex_unlock (&mdev->graph_mutex); -return 0 ; +return ret; }EXPORT_SYMBOL_GPL (media_entity_pads_init);
Signature · 3 lines
Regards, Laurent Pinchart
Sakari Ailus
maintainerRe: [RFC 4/7] media: mc: Check pad flag validity
Quoted 23 lines · Laurent Pinchart
Hi Sakari,
Thank you for the patch.
Quoted 18 lines · Sakari Ailus
Check the validity of pad flags on entity init. Exactly one of the flags must be set.
Signed-off-by Sakari Ailus <[email protected]>--- drivers/media/mc/mc-entity.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)drivers/media/mc/mc-entity.c +1
-0@@ -197,6 +197,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,struct media_device *mdev = entity->graph_obj.mdev;struct media_pad *iter;unsigned int i =0 ; +int ret =0 ;
Quoted 15 lines
drivers/media/mc/mc-entity.c int media_entity_pads_init(struct media_entity *entity, u16 num_pads, · line 218
Quoted 11 lines
drivers/media/mc/mc-entity.c +5
-0if (num_pads >=MEDIA_ENTITY_MAX_PADS )return -E2BIG ;@@ -210,6 +211,15 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,media_entity_for_each_pad (entity, iter) { iter->entity = entity; iter->index = i++; + +if (hweight32 (iter->flags & (MEDIA_PAD_FL_SINK | +MEDIA_PAD_FL_SOURCE | +MEDIA_PAD_FL_INTERNAL_SOURCE )) + !=1 ) {I would add this check as patch 1/7 in this series, with
FL_SINKanFL_SOURCEonly, and then extend it in the patch where you introduceFL_INTERNAL_SOURCE.
Sounds reasonable.
Quoted 22 lines
Quoted 17 lines
drivers/media/mc/mc-entity.c +5
-1+ ret = -EINVAL ; +break ; + } +if (mdev)media_gobj_create (mdev,MEDIA_GRAPH_PAD , &iter->graph_obj);@@ -218,7 +228,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,if (mdev)mutex_unlock (&mdev->graph_mutex); -return 0 ; +return ret; }EXPORT_SYMBOL_GPL (media_entity_pads_init);Signature · 3 lines
Regards, Laurent Pinchart
Signature · 1 line
Sakari Ailus
Sakari Ailus
maintainerRFC 5/7 media: uapi: Add generic serial metadata mbus formats
In reply to Sakari AilusAdd generic serial metadata mbus formats. These formats describe data width and packing but not the content itself. The reason for specifying such formats is that the formats as such are fairly device specific but they are still handled by CSI-2 receiver drivers that should not be aware of device specific formats. What makes generic metadata formats possible is that these formats are parsed by software only, after capturing the data to system memory.
--- .../media/v4l/subdev-formats.rst | 257 ++++++++++++++++++ include/uapi/linux/media-bus-format.h | 9 + 2 files changed, 266 insertions(+)
Documentation/userspace-api/media/v4l/subdev-formats.rst +257 -0
@@ -8234,3 +8234,260 @@ The following table lists the existing metadata formats. both sides of the link and the bus format is a fixed metadata format that is not configurable from userspace. Width and height will be set to 0 for this format. + +Generic Serial Metadata Formats +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Generic serial metadata formats are used on serial busses where the actual data +content is more or less device specific but the data is transmitted and received +by multiple devices that do not process the data in any way, simply writing +it to system memory for processing in software at the end of the pipeline. + +The more specific variant describing the actual data is used on the internal +source pad of the originating sub-device. + +"b" in an array cell signifies a byte of data, followed by the number of byte +and finally the bit number in subscript. "p" indicates a padding bit. + +.. _media-bus-format-generic-meta: + +.. cssclass: longtable + +.. flat-table:: Generic Serial Metadata Formats + :header-rows: 2 + :stub-columns: 0 + + * - Identifier + - Code + - + - :cspan:`23` Data organization + * - + - + - Bit + - 23 + - 22 + - 21 + - 20 + - 19 + - 18 + - 17 + - 16 + - 15 + - 14 + - 13 + - 12 + - 11 + - 10 + - 9 + - 8 + - 7 + - 6 + - 5 + - 4 + - 3 + - 2 + - 1 + - 0 + * .. _MEDIA-BUS-FMT-META-1X8-8: + + - MEDIA_BUS_FMT_META_1X8_8 + - 0x8001 + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + * .. _MEDIA-BUS-FMT-META-1X8-10: + + - MEDIA_BUS_FMT_META_1X8_10 + - 0x8002 + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-12: + + - MEDIA_BUS_FMT_META_1X8_12 + - 0x8003 + - + - + - + - + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-14: + + - MEDIA_BUS_FMT_META_1X8_14 + - 0x8004 + - + - + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-16: + + - MEDIA_BUS_FMT_META_1X8_16 + - 0x8005 + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + - p + - p + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-20: + + - MEDIA_BUS_FMT_META_1X8_20 + - 0x8007 + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-24: + + - MEDIA_BUS_FMT_META_1X8_24 + - 0x8009 + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p
include/uapi/linux/media-bus-format.h +9 -0
@@ -173,4 +173,13 @@*/ #define MEDIA_BUS_FMT_METADATA_FIXED 0x7001 +/* Generic line based metadata formats for serial buses. Next is 0x800b. */ +#define MEDIA_BUS_FMT_META_1X8_8 0x8001 +#define MEDIA_BUS_FMT_META_1X8_10 0x8002 +#define MEDIA_BUS_FMT_META_1X8_12 0x8003 +#define MEDIA_BUS_FMT_META_1X8_14 0x8004 +#define MEDIA_BUS_FMT_META_1X8_16 0x8005 +#define MEDIA_BUS_FMT_META_1X8_20 0x8007 +#define MEDIA_BUS_FMT_META_1X8_24 0x8009 +#endif /* __LINUX_MEDIA_BUS_FORMAT_H */
Signature · 1 line
2.30.2
Laurent Pinchart
maintainerRe: [RFC 5/7] media: uapi: Add generic serial metadata mbus formats
Hi Sakari,
Thank you for the patch.
Quoted 19 lines · Sakari Ailus
Add generic serial metadata mbus formats. These formats describe data width and packing but not the content itself. The reason for specifying such formats is that the formats as such are fairly device specific but they are still handled by CSI-2 receiver drivers that should not be aware of device specific formats. What makes generic metadata formats possible is that these formats are parsed by software only, after capturing the data to system memory.
Signed-off-by Sakari Ailus <[email protected]>--- .../media/v4l/subdev-formats.rst | 257 ++++++++++++++++++ include/uapi/linux/media-bus-format.h | 9 + 2 files changed, 266 insertions(+)Documentation/userspace-api/media/v4l/subdev-formats.rst +0
-0
Documentation/userspace-api/media/v4l/subdev-formats.rst The following table lists the existing metadata formats. · line 8241
Quoted 8 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +5
-0@@ -8234,3 +8234,260 @@ The following table lists the existing metadata formats. both sides of the link and the bus format is a fixed metadata format that is not configurable from userspace. Width and height will be set to 0 for this format. + +Generic Serial Metadata Formats +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Generic serial metadata formats are used on serial busses where the actual data
s/busses/buses/
Documentation/userspace-api/media/v4l/subdev-formats.rst The following table lists the existing metadata formats. · line 8247
Quoted 6 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +6
-0+content is more or less device specific but the data is transmitted and received +by multiple devices that do not process the data in any way, simply writing +it to system memory for processing in software at the end of the pipeline. + +The more specific variant describing the actual data is used on the internal +source pad of the originating sub-device.
What do you mean by "more specific variant" here ? Please include an example in the documentation.
I'm not sure I like mentioning internal source pads here, are we guaranteed that metadata will always originate from an internal source pad ?
Quoted 252 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +246
-0+ +"b" in an array cell signifies a byte of data, followed by the number of byte +and finally the bit number in subscript. "p" indicates a padding bit. + +.. _media-bus-format-generic-meta: + +.. cssclass: longtable + +.. flat-table:: Generic Serial Metadata Formats + :header-rows: 2 + :stub-columns: 0 + + * - Identifier + - Code + - + - :cspan:`23` Data organization + * - + - + - Bit + - 23 + - 22 + - 21 + - 20 + - 19 + - 18 + - 17 + - 16 + - 15 + - 14 + - 13 + - 12 + - 11 + - 10 + - 9 + - 8 + - 7 + - 6 + - 5 + - 4 + - 3 + - 2 + - 1 + - 0 + * .. _MEDIA-BUS-FMT-META-1X8-8: + + - MEDIA_BUS_FMT_META_1X8_8 + - 0x8001 + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + * .. _MEDIA-BUS-FMT-META-1X8-10: + + - MEDIA_BUS_FMT_META_1X8_10 + - 0x8002 + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-12: + + - MEDIA_BUS_FMT_META_1X8_12 + - 0x8003 + - + - + - + - + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-14: + + - MEDIA_BUS_FMT_META_1X8_14 + - 0x8004 + - + - + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-16: + + - MEDIA_BUS_FMT_META_1X8_16 + - 0x8005 + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + - p + - p + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-20: + + - MEDIA_BUS_FMT_META_1X8_20 + - 0x8007 + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-24: + + - MEDIA_BUS_FMT_META_1X8_24 + - 0x8009 + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - pinclude/uapi/linux/media-bus-format.h +0
-0@@ -173,4 +173,13 @@*/
include/uapi/linux/media-bus-format.h line 183
Quoted 10 lines
include/uapi/linux/media-bus-format.h +8
-0#define MEDIA_BUS_FMT_METADATA_FIXED 0x7001 +/* Generic line based metadata formats for serial buses. Next is 0x800b. */ +#define MEDIA_BUS_FMT_META_1X8_8 0x8001 +#define MEDIA_BUS_FMT_META_1X8_10 0x8002 +#define MEDIA_BUS_FMT_META_1X8_12 0x8003 +#define MEDIA_BUS_FMT_META_1X8_14 0x8004 +#define MEDIA_BUS_FMT_META_1X8_16 0x8005 +#define MEDIA_BUS_FMT_META_1X8_20 0x8007 +#define MEDIA_BUS_FMT_META_1X8_24 0x8009
We've discussed this before privately, it's time to come to a conclusion :-)
My preference would be
#define MEDIA_BUS_FMT_META_RAW8 0x8001
#define MEDIA_BUS_FMT_META_RAW10 0x8002
#define MEDIA_BUS_FMT_META_RAW12 0x8003
#define MEDIA_BUS_FMT_META_RAW14 0x8004
#define MEDIA_BUS_FMT_META_RAW16 0x8005
#define MEDIA_BUS_FMT_META_RAW20 0x8007
#define MEDIA_BUS_FMT_META_RAW24 0x8009
without defining the contents of the data (that is, no mention of padding bits).
Quoted 2 lines
include/uapi/linux/media-bus-format.h +1
-0+#endif /* __LINUX_MEDIA_BUS_FORMAT_H */
Signature · 3 lines
Regards, Laurent Pinchart
Sakari Ailus
maintainerRe: [RFC 5/7] media: uapi: Add generic serial metadata mbus formats
Hi Laurent,
Quoted 45 lines · Laurent Pinchart
Hi Sakari,
Thank you for the patch.
Quoted 19 lines · Sakari Ailus
Add generic serial metadata mbus formats. These formats describe data width and packing but not the content itself. The reason for specifying such formats is that the formats as such are fairly device specific but they are still handled by CSI-2 receiver drivers that should not be aware of device specific formats. What makes generic metadata formats possible is that these formats are parsed by software only, after capturing the data to system memory.
Signed-off-by Sakari Ailus <[email protected]>--- .../media/v4l/subdev-formats.rst | 257 ++++++++++++++++++ include/uapi/linux/media-bus-format.h | 9 + 2 files changed, 266 insertions(+)Documentation/userspace-api/media/v4l/subdev-formats.rst +0
-0Documentation/userspace-api/media/v4l/subdev-formats.rst The following table lists the existing metadata formats. · line 8241
Quoted 8 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +5
-0@@ -8234,3 +8234,260 @@ The following table lists the existing metadata formats. both sides of the link and the bus format is a fixed metadata format that is not configurable from userspace. Width and height will be set to 0 for this format. + +Generic Serial Metadata Formats +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Generic serial metadata formats are used on serial busses where the actual datas/busses/buses/
Documentation/userspace-api/media/v4l/subdev-formats.rst The following table lists the existing metadata formats. · line 8247
Quoted 6 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +6
-0+content is more or less device specific but the data is transmitted and received +by multiple devices that do not process the data in any way, simply writing +it to system memory for processing in software at the end of the pipeline. + +The more specific variant describing the actual data is used on the internal +source pad of the originating sub-device.What do you mean by "more specific variant" here ? Please include an example in the documentation.
Quoted 3 lines
I'm not sure I like mentioning internal source pads here, are we guaranteed that metadata will always originate from an internal source pad ?
We'll have to, otherwise the specific format will be visible on the link, and link validator on the sink end won't (and shouldn't) know about it (and that would systematically lead to a failure to validate it).
Quoted 277 lines
Quoted 253 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +246
-0+ +"b" in an array cell signifies a byte of data, followed by the number of byte +and finally the bit number in subscript. "p" indicates a padding bit. + +.. _media-bus-format-generic-meta: + +.. cssclass: longtable + +.. flat-table:: Generic Serial Metadata Formats + :header-rows: 2 + :stub-columns: 0 + + * - Identifier + - Code + - + - :cspan:`23` Data organization + * - + - + - Bit + - 23 + - 22 + - 21 + - 20 + - 19 + - 18 + - 17 + - 16 + - 15 + - 14 + - 13 + - 12 + - 11 + - 10 + - 9 + - 8 + - 7 + - 6 + - 5 + - 4 + - 3 + - 2 + - 1 + - 0 + * .. _MEDIA-BUS-FMT-META-1X8-8: + + - MEDIA_BUS_FMT_META_1X8_8 + - 0x8001 + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + * .. _MEDIA-BUS-FMT-META-1X8-10: + + - MEDIA_BUS_FMT_META_1X8_10 + - 0x8002 + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-12: + + - MEDIA_BUS_FMT_META_1X8_12 + - 0x8003 + - + - + - + - + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-14: + + - MEDIA_BUS_FMT_META_1X8_14 + - 0x8004 + - + - + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-16: + + - MEDIA_BUS_FMT_META_1X8_16 + - 0x8005 + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + - p + - p + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-20: + + - MEDIA_BUS_FMT_META_1X8_20 + - 0x8007 + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-24: + + - MEDIA_BUS_FMT_META_1X8_24 + - 0x8009 + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - pinclude/uapi/linux/media-bus-format.h +0
-0@@ -173,4 +173,13 @@*/ include/uapi/linux/media-bus-format.h line 183
Quoted 10 lines
include/uapi/linux/media-bus-format.h +8
-0#define MEDIA_BUS_FMT_METADATA_FIXED 0x7001 +/* Generic line based metadata formats for serial buses. Next is 0x800b. */ +#define MEDIA_BUS_FMT_META_1X8_8 0x8001 +#define MEDIA_BUS_FMT_META_1X8_10 0x8002 +#define MEDIA_BUS_FMT_META_1X8_12 0x8003 +#define MEDIA_BUS_FMT_META_1X8_14 0x8004 +#define MEDIA_BUS_FMT_META_1X8_16 0x8005 +#define MEDIA_BUS_FMT_META_1X8_20 0x8007 +#define MEDIA_BUS_FMT_META_1X8_24 0x8009We've discussed this before privately, it's time to come to a conclusion :-)
My preference would be
#define
MEDIA_BUS_FMT_META_RAW80x8001
#defineMEDIA_BUS_FMT_META_RAW100x8002
#defineMEDIA_BUS_FMT_META_RAW120x8003
#defineMEDIA_BUS_FMT_META_RAW140x8004
#defineMEDIA_BUS_FMT_META_RAW160x8005
#defineMEDIA_BUS_FMT_META_RAW200x8007
#defineMEDIA_BUS_FMT_META_RAW240x8009
Quoted 2 lines
without defining the contents of the data (that is, no mention of padding bits).
I don't object but I'd like to know what others think.
If we ever need formats with padding (or other depths than 8) we'll need to change the naming. This may still be unlikely.
Signature · 3 lines
Regards, Sakari Ailus
Re: [RFC 5/7] media: uapi: Add generic serial metadata mbus formats
In reply to Sakari AilusQuoted 23 lines · Sakari Ailus
Add generic serial metadata mbus formats. These formats describe data width and packing but not the content itself. The reason for specifying such formats is that the formats as such are fairly device specific but they are still handled by CSI-2 receiver drivers that should not be aware of device specific formats. What makes generic metadata formats possible is that these formats are parsed by software only, after capturing the data to system memory.
Signed-off-by Sakari Ailus <[email protected]>--- .../media/v4l/subdev-formats.rst | 257 ++++++++++++++++++ include/uapi/linux/media-bus-format.h | 9 + 2 files changed, 266 insertions(+)Documentation/userspace-api/media/v4l/subdev-formats.rst +1
-0@@ -8234,3 +8234,260 @@ The following table lists the existing metadata formats. both sides of the link and the bus format is a fixed metadata format that is not configurable from userspace. Width and height will be set to 0 for this format. +
Documentation/userspace-api/media/v4l/subdev-formats.rst The following table lists the existing metadata formats. · line 8247
Quoted 10 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +10
-0+Generic Serial Metadata Formats +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Generic serial metadata formats are used on serial busses where the actual data +content is more or less device specific but the data is transmitted and received +by multiple devices that do not process the data in any way, simply writing +it to system memory for processing in software at the end of the pipeline. + +The more specific variant describing the actual data is used on the internal +source pad of the originating sub-device.
As Laurent said as well, the mention of 'internal source pad' is confusing.
Aren't there two possibilities here? Either the entity has just a source pad that delivers the metadata, and that source pad has the precise metadata format. Or the entity supports routing and then there is a sink pad connected to an internal source that is routed to a source pad. In that case that sink pad has the precise metadata format.
Apologies if I am wrong, it's hard to keep track of all the possibilities, especially if you do not use this regularly.
Quoted 251 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +246
-0+ +"b" in an array cell signifies a byte of data, followed by the number of byte +and finally the bit number in subscript. "p" indicates a padding bit. + +.. _media-bus-format-generic-meta: + +.. cssclass: longtable + +.. flat-table:: Generic Serial Metadata Formats + :header-rows: 2 + :stub-columns: 0 + + * - Identifier + - Code + - + - :cspan:`23` Data organization + * - + - + - Bit + - 23 + - 22 + - 21 + - 20 + - 19 + - 18 + - 17 + - 16 + - 15 + - 14 + - 13 + - 12 + - 11 + - 10 + - 9 + - 8 + - 7 + - 6 + - 5 + - 4 + - 3 + - 2 + - 1 + - 0 + * .. _MEDIA-BUS-FMT-META-1X8-8: + + - MEDIA_BUS_FMT_META_1X8_8 + - 0x8001 + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + * .. _MEDIA-BUS-FMT-META-1X8-10: + + - MEDIA_BUS_FMT_META_1X8_10 + - 0x8002 + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-12: + + - MEDIA_BUS_FMT_META_1X8_12 + - 0x8003 + - + - + - + - + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-14: + + - MEDIA_BUS_FMT_META_1X8_14 + - 0x8004 + - + - + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-16: + + - MEDIA_BUS_FMT_META_1X8_16 + - 0x8005 + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + - p + - p + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-20: + + - MEDIA_BUS_FMT_META_1X8_20 + - 0x8007 + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-24: + + - MEDIA_BUS_FMT_META_1X8_24 + - 0x8009 + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - pinclude/uapi/linux/media-bus-format.h +0
-0
include/uapi/linux/media-bus-format.h line 181
Quoted 9 lines
include/uapi/linux/media-bus-format.h +6
-0@@ -173,4 +173,13 @@*/ #define MEDIA_BUS_FMT_METADATA_FIXED 0x7001 +/* Generic line based metadata formats for serial buses. Next is 0x800b. */ +#define MEDIA_BUS_FMT_META_1X8_8 0x8001 +#define MEDIA_BUS_FMT_META_1X8_10 0x8002 +#define MEDIA_BUS_FMT_META_1X8_12 0x8003 +#define MEDIA_BUS_FMT_META_1X8_14 0x8004 +#define MEDIA_BUS_FMT_META_1X8_16 0x8005
You skip 0x8006 here in case there will be a _18 variant?
include/uapi/linux/media-bus-format.h line 185
Quoted 4 lines
include/uapi/linux/media-bus-format.h +3
-0+#define MEDIA_BUS_FMT_META_1X8_20 0x8007 +#define MEDIA_BUS_FMT_META_1X8_24 0x8009 +#endif /* __LINUX_MEDIA_BUS_FORMAT_H */
Regards,
Hans
Sakari Ailus
maintainerRe: [RFC 5/7] media: uapi: Add generic serial metadata mbus formats
Hi Hans,
Quoted 37 lines · Hans Verkuil
Quoted 23 lines · Sakari Ailus
Add generic serial metadata mbus formats. These formats describe data width and packing but not the content itself. The reason for specifying such formats is that the formats as such are fairly device specific but they are still handled by CSI-2 receiver drivers that should not be aware of device specific formats. What makes generic metadata formats possible is that these formats are parsed by software only, after capturing the data to system memory.
Signed-off-by Sakari Ailus <[email protected]>--- .../media/v4l/subdev-formats.rst | 257 ++++++++++++++++++ include/uapi/linux/media-bus-format.h | 9 + 2 files changed, 266 insertions(+)Documentation/userspace-api/media/v4l/subdev-formats.rst +1
-0@@ -8234,3 +8234,260 @@ The following table lists the existing metadata formats. both sides of the link and the bus format is a fixed metadata format that is not configurable from userspace. Width and height will be set to 0 for this format. +Documentation/userspace-api/media/v4l/subdev-formats.rst The following table lists the existing metadata formats. · line 8247
Quoted 10 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +10
-0+Generic Serial Metadata Formats +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Generic serial metadata formats are used on serial busses where the actual data +content is more or less device specific but the data is transmitted and received +by multiple devices that do not process the data in any way, simply writing +it to system memory for processing in software at the end of the pipeline. + +The more specific variant describing the actual data is used on the internal +source pad of the originating sub-device.As Laurent said as well, the mention of 'internal source pad' is confusing.
Quoted 8 lines
Aren't there two possibilities here? Either the entity has just a source pad that delivers the metadata, and that source pad has the precise metadata format. Or the entity supports routing and then there is a sink pad connected to an internal source that is routed to a source pad. In that case that sink pad has the precise metadata format.
Apologies if I am wrong, it's hard to keep track of all the possibilities, especially if you do not use this regularly.
It's required the generic metadata format is present only on internal pads, otherwise link validation will fail --- the next device won't be aware of the specific mbus code.
Quoted 255 lines
Quoted 255 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +246
-0+ +"b" in an array cell signifies a byte of data, followed by the number of byte +and finally the bit number in subscript. "p" indicates a padding bit. + +.. _media-bus-format-generic-meta: + +.. cssclass: longtable + +.. flat-table:: Generic Serial Metadata Formats + :header-rows: 2 + :stub-columns: 0 + + * - Identifier + - Code + - + - :cspan:`23` Data organization + * - + - + - Bit + - 23 + - 22 + - 21 + - 20 + - 19 + - 18 + - 17 + - 16 + - 15 + - 14 + - 13 + - 12 + - 11 + - 10 + - 9 + - 8 + - 7 + - 6 + - 5 + - 4 + - 3 + - 2 + - 1 + - 0 + * .. _MEDIA-BUS-FMT-META-1X8-8: + + - MEDIA_BUS_FMT_META_1X8_8 + - 0x8001 + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + * .. _MEDIA-BUS-FMT-META-1X8-10: + + - MEDIA_BUS_FMT_META_1X8_10 + - 0x8002 + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-12: + + - MEDIA_BUS_FMT_META_1X8_12 + - 0x8003 + - + - + - + - + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-14: + + - MEDIA_BUS_FMT_META_1X8_14 + - 0x8004 + - + - + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-16: + + - MEDIA_BUS_FMT_META_1X8_16 + - 0x8005 + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + - p + - p + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-20: + + - MEDIA_BUS_FMT_META_1X8_20 + - 0x8007 + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-24: + + - MEDIA_BUS_FMT_META_1X8_24 + - 0x8009 + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - pinclude/uapi/linux/media-bus-format.h +0
-0@@ -173,4 +173,13 @@*/ #define MEDIA_BUS_FMT_METADATA_FIXED 0x7001
Quoted 8 lines
include/uapi/linux/media-bus-format.h line 181
Quoted 6 lines
include/uapi/linux/media-bus-format.h +6
-0+/* Generic line based metadata formats for serial buses. Next is 0x800b. */ +#define MEDIA_BUS_FMT_META_1X8_8 0x8001 +#define MEDIA_BUS_FMT_META_1X8_10 0x8002 +#define MEDIA_BUS_FMT_META_1X8_12 0x8003 +#define MEDIA_BUS_FMT_META_1X8_14 0x8004 +#define MEDIA_BUS_FMT_META_1X8_16 0x8005You skip
0x8006here in case there will be a _18 variant?
I removed duplicate entries but forgot to change the values. I'll address that for v2.
There are different packings defined for these formats, sensor generally using the pixel data equivalent format. With 16 and more bits per sample, multiple octets of metadata can be transported in that sample, and those formats are equivalent to the formats at half (or third) of the sample size. Say, alternative format for 16 bpp image data to 1X8_16 is 1X8_8.
Signature · 3 lines
Kind regards, Sakari Ailus
Re: [RFC 5/7] media: uapi: Add generic serial metadata mbus formats
In reply to Sakari AilusQuoted 23 lines · Sakari Ailus
Add generic serial metadata mbus formats. These formats describe data width and packing but not the content itself. The reason for specifying such formats is that the formats as such are fairly device specific but they are still handled by CSI-2 receiver drivers that should not be aware of device specific formats. What makes generic metadata formats possible is that these formats are parsed by software only, after capturing the data to system memory.
Signed-off-by Sakari Ailus <[email protected]>--- .../media/v4l/subdev-formats.rst | 257 ++++++++++++++++++ include/uapi/linux/media-bus-format.h | 9 + 2 files changed, 266 insertions(+)Documentation/userspace-api/media/v4l/subdev-formats.rst +1
-0@@ -8234,3 +8234,260 @@ The following table lists the existing metadata formats. both sides of the link and the bus format is a fixed metadata format that is not configurable from userspace. Width and height will be set to 0 for this format. +
Documentation/userspace-api/media/v4l/subdev-formats.rst The following table lists the existing metadata formats. · line 8247
Quoted 10 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +10
-0+Generic Serial Metadata Formats +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Generic serial metadata formats are used on serial busses where the actual data +content is more or less device specific but the data is transmitted and received +by multiple devices that do not process the data in any way, simply writing +it to system memory for processing in software at the end of the pipeline. + +The more specific variant describing the actual data is used on the internal +source pad of the originating sub-device.
I'm a bit confused here: on which pads are these generic mbus formats set? And who sets this?
I.e. the internal source pad has e.g. MEDIA_BUS_FMT_META_FOOBAR_HIST as specific format. It's routed to a source pad + stream pair for transfer over CSI. Does that source pad + stream pair use the generic mediabus format?
Regards,
Hans
Quoted 264 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +246
-0+ +"b" in an array cell signifies a byte of data, followed by the number of byte +and finally the bit number in subscript. "p" indicates a padding bit. + +.. _media-bus-format-generic-meta: + +.. cssclass: longtable + +.. flat-table:: Generic Serial Metadata Formats + :header-rows: 2 + :stub-columns: 0 + + * - Identifier + - Code + - + - :cspan:`23` Data organization + * - + - + - Bit + - 23 + - 22 + - 21 + - 20 + - 19 + - 18 + - 17 + - 16 + - 15 + - 14 + - 13 + - 12 + - 11 + - 10 + - 9 + - 8 + - 7 + - 6 + - 5 + - 4 + - 3 + - 2 + - 1 + - 0 + * .. _MEDIA-BUS-FMT-META-1X8-8: + + - MEDIA_BUS_FMT_META_1X8_8 + - 0x8001 + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + * .. _MEDIA-BUS-FMT-META-1X8-10: + + - MEDIA_BUS_FMT_META_1X8_10 + - 0x8002 + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-12: + + - MEDIA_BUS_FMT_META_1X8_12 + - 0x8003 + - + - + - + - + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-14: + + - MEDIA_BUS_FMT_META_1X8_14 + - 0x8004 + - + - + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-16: + + - MEDIA_BUS_FMT_META_1X8_16 + - 0x8005 + - + - + - + - + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + - p + - p + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-20: + + - MEDIA_BUS_FMT_META_1X8_20 + - 0x8007 + - + - + - + - + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + * .. _MEDIA-BUS-FMT-META-1X8-24: + + - MEDIA_BUS_FMT_META_1X8_24 + - 0x8009 + - + - b0\ :sub:`7` + - b0\ :sub:`6` + - b0\ :sub:`5` + - b0\ :sub:`4` + - b0\ :sub:`3` + - b0\ :sub:`2` + - b0\ :sub:`1` + - b0\ :sub:`0` + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - p + - pinclude/uapi/linux/media-bus-format.h +9
-0@@ -173,4 +173,13 @@*/ #define MEDIA_BUS_FMT_METADATA_FIXED 0x7001 +/* Generic line based metadata formats for serial buses. Next is 0x800b. */ +#define MEDIA_BUS_FMT_META_1X8_8 0x8001 +#define MEDIA_BUS_FMT_META_1X8_10 0x8002 +#define MEDIA_BUS_FMT_META_1X8_12 0x8003 +#define MEDIA_BUS_FMT_META_1X8_14 0x8004 +#define MEDIA_BUS_FMT_META_1X8_16 0x8005 +#define MEDIA_BUS_FMT_META_1X8_20 0x8007 +#define MEDIA_BUS_FMT_META_1X8_24 0x8009 +#endif /* __LINUX_MEDIA_BUS_FORMAT_H */
Sakari Ailus
maintainerRe: [RFC 5/7] media: uapi: Add generic serial metadata mbus formats
Hi Hans,
Quoted 29 lines · Hans Verkuil
Quoted 28 lines · Sakari Ailus
Add generic serial metadata mbus formats. These formats describe data width and packing but not the content itself. The reason for specifying such formats is that the formats as such are fairly device specific but they are still handled by CSI-2 receiver drivers that should not be aware of device specific formats. What makes generic metadata formats possible is that these formats are parsed by software only, after capturing the data to system memory.
Signed-off-by Sakari Ailus <[email protected]>--- .../media/v4l/subdev-formats.rst | 257 ++++++++++++++++++ include/uapi/linux/media-bus-format.h | 9 + 2 files changed, 266 insertions(+)Documentation/userspace-api/media/v4l/subdev-formats.rst +6
-0@@ -8234,3 +8234,260 @@ The following table lists the existing metadata formats. both sides of the link and the bus format is a fixed metadata format that is not configurable from userspace. Width and height will be set to 0 for this format. + +Generic Serial Metadata Formats +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Generic serial metadata formats are used on serial busses where the actual data +content is more or less device specific but the data is transmitted and received
Quoted 8 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst The following table lists the existing metadata formats. · line 8247
Quoted 5 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +5
-0+by multiple devices that do not process the data in any way, simply writing +it to system memory for processing in software at the end of the pipeline. + +The more specific variant describing the actual data is used on the internal +source pad of the originating sub-device.I'm a bit confused here: on which pads are these generic mbus formats set? And who sets this?
As with other pads, the user can set the format. In most cases only a single option is available whenever the device ourputs metadata over a stream (and thus uses on of the generic line based metadata formats).
Quoted 5 lines
I.e. the internal source pad has e.g.
MEDIA_BUS_FMT_META_FOOBAR_HISTas specific format. It's routed to a source pad + stream pair for transfer over CSI. Does that source pad + stream pair use the generic mediabus format?
Correct.
Signature · 1 line
Sakari Ailus
Sakari Ailus
maintainerRFC 6/7 media: uapi: Add generic 8-bit metadata format definitions
In reply to Sakari AilusGeneric 8-bit metadata formats define the in-memory data layout but not the format of the data itself. The reasoning for having such formats is to allow CSI-2 receiver drivers to receive and DMA drivers to write the data to memory without knowing a large number of device specific formats.
These formats may be used only in conjunction of a Media controller pipeline where the internal pad of the source sub-device defines the specific format of the data (using an mbus code).
--- .../userspace-api/media/v4l/meta-formats.rst | 1 + .../media/v4l/metafmt-generic.rst | 317 ++++++++++++++++++ drivers/media/v4l2-core/v4l2-ioctl.c | 8 + include/uapi/linux/videodev2.h | 9 + 4 files changed, 335 insertions(+) create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
Documentation/userspace-api/media/v4l/meta-formats.rst +1 -0
@@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only. metafmt-vsp1-hgo metafmt-vsp1-hgt metafmt-vivid + metafmt-generic
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +317 -0
317 lines not on this page · raw message
drivers/media/v4l2-core/v4l2-ioctl.c +8 -0
@@ -1451,6 +1451,14 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)case V4L2_PIX_FMT_Y210 : descr ="10-bit YUYV Packed" ;break ;case V4L2_PIX_FMT_Y212 : descr ="12-bit YUYV Packed" ;break ;case V4L2_PIX_FMT_Y216 : descr ="16-bit YUYV Packed" ;break ; +case V4L2_META_FMT_GENERIC_8 : descr ="8-bit Generic Metadata" ;break ; +case V4L2_META_FMT_GENERIC_CSI2_10 : descr ="8b Generic Meta, 10b CSI-2" ;break ; +case V4L2_META_FMT_GENERIC_CSI2_12 : descr ="8b Generic Meta, 12b CSI-2" ;break ; +case V4L2_META_FMT_GENERIC_CSI2_14 : descr ="8b Generic Meta, 14b CSI-2" ;break ; +case V4L2_META_FMT_GENERIC_CSI2_16 : descr ="8b Generic Meta, 16b CSI-2" ;break ; +case V4L2_META_FMT_GENERIC_CSI2_20 : descr ="8b Generic Meta, 20b CSI-2" ;break ; +case V4L2_META_FMT_GENERIC_CSI2_24 : descr ="8b Generic Meta, 24b CSI-2" ;break ; +case V4L2_META_FMT_GENERIC_CSI2_2_24 : descr ="2x8b Generic Meta, 24b CSI-2" ;break ;default :/* Compressed formats */
include/uapi/linux/videodev2.h +9 -0
@@ -834,6 +834,15 @@ struct v4l2_pix_format {#define V4L2_META_FMT_RK_ISP1_PARAMS v4l2_fourcc('R', 'K', '1', 'P')/* Rockchip ISP1 3A Parameters */ #define V4L2_META_FMT_RK_ISP1_STAT_3A v4l2_fourcc('R', 'K', '1', 'S')/* Rockchip ISP1 3A Statistics */ +#define V4L2_META_FMT_GENERIC_8 v4l2_fourcc('M', 'E', 'T', '8')/* Generic 8-bit metadata */ +#define V4L2_META_FMT_GENERIC_CSI2_10 v4l2_fourcc('M', 'C', '1', 'A')/* 10-bit CSI-2 packed 8-bit metadata */ +#define V4L2_META_FMT_GENERIC_CSI2_12 v4l2_fourcc('M', 'C', '1', 'C')/* 12-bit CSI-2 packed 8-bit metadata */ +#define V4L2_META_FMT_GENERIC_CSI2_14 v4l2_fourcc('M', 'C', '1', 'E')/* 14-bit CSI-2 packed 8-bit metadata */ +#define V4L2_META_FMT_GENERIC_CSI2_16 v4l2_fourcc('M', 'C', '1', 'G')/* 16-bit CSI-2 packed 8-bit metadata */ +#define V4L2_META_FMT_GENERIC_CSI2_20 v4l2_fourcc('M', 'C', '1', 'K')/* 20-bit CSI-2 packed 8-bit metadata */ +#define V4L2_META_FMT_GENERIC_CSI2_24 v4l2_fourcc('M', 'C', '1', 'O')/* 24-bit CSI-2 packed 8-bit metadata */ +#define V4L2_META_FMT_GENERIC_CSI2_2_24 v4l2_fourcc('M', 'C', '2', 'O')/* 2 bytes of 8-bit metadata, 24-bit CSI-2 packed */ +/* priv field value to indicates that subsequent fields are valid. */ #define V4L2_PIX_FMT_PRIV_MAGIC 0xfeedcafe
Signature · 1 line
2.30.2
Re: [RFC 6/7] media: uapi: Add generic 8-bit metadata format definitions
Quoted 50 lines · Sakari Ailus
Generic 8-bit metadata formats define the in-memory data layout but not the format of the data itself. The reasoning for having such formats is to allow CSI-2 receiver drivers to receive and DMA drivers to write the data to memory without knowing a large number of device specific formats.
These formats may be used only in conjunction of a Media controller pipeline where the internal pad of the source sub-device defines the specific format of the data (using an mbus code).
Signed-off-by Sakari Ailus <[email protected]>--- .../userspace-api/media/v4l/meta-formats.rst | 1 + .../media/v4l/metafmt-generic.rst | 317 ++++++++++++++++++ drivers/media/v4l2-core/v4l2-ioctl.c | 8 + include/uapi/linux/videodev2.h | 9 + 4 files changed, 335 insertions(+) create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rstDocumentation/userspace-api/media/v4l/meta-formats.rst +1
-0@@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only. metafmt-vsp1-hgo metafmt-vsp1-hgt metafmt-vivid + metafmt-genericDocumentation/userspace-api/media/v4l/metafmt-generic.rst new file +17
-0@@ -0,0 +1,317 @@ +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later + +************************************************************************************************************************************************************************************************************************************************************************************************************************** +V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O') +************************************************************************************************************************************************************************************************************************************************************************************************************************** + + +Generic line-based metadata formats + + +Description +=========== + +These generic line-based metadata formats define the memory layout of the data +without defining the format or meaning of the metadata itself. These formats may +only be used with a Media controller pipeline where the more specific format is +defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL-SOURCE>` of the
Documentation/userspace-api/media/v4l/metafmt-generic.rst line 27
Quoted 10 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +10
-0+source sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`. + +.. _v4l2-meta-fmt-generic-8: + +V4L2_META_FMT_GENERIC_8 +----------------------- + +The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format whereas the +rest of the formats have CSI-2 packing that are typically used on camera sensors +in conjunction with pixel data of the same bit depth.
You keep referring to CSI-2 packing without ever defining it.
I'm not sure you should include CSI2 in the name either. It is OK to say something here like: "The way the data is stored follows the CSI-2 specification."
But it is not limited to CSI-2 and can be used with other buses as well.
Documentation/userspace-api/media/v4l/metafmt-generic.rst line 30
Quoted 3 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +3
-0+ +This format is also used on CSI-2 on both 8 bits per sample as well as on +16 bits per sample when two bytes of metadata are packed into one sample.
Little or big endian?
Regards,
Hans
Quoted 326 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +287
-0+ +**Byte Order Of V4L2_META_FMT_GENERIC_8.** +Each cell is one byte. "M" denotes a byte of metadata. + +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}| + +.. flat-table:: + :header-rows: 0 + :stub-columns: 0 + :widths: 12 8 8 8 8 + + * - start + 0: + - M\ :sub:`00` + - M\ :sub:`10` + - M\ :sub:`20` + - M\ :sub:`30` + * - start + 4: + - M\ :sub:`01` + - M\ :sub:`11` + - M\ :sub:`21` + - M\ :sub:`31` + +.. _v4l2-meta-fmt-generic-csi2-10: + +V4L2_META_FMT_GENERIC_CSI2_10 +----------------------------- + +V4L2_META_FMT_GENERIC_CSI2_10 contains 8-bit generic metadata with CSI-2 +packing, 10 bits for each 8 bits of data. Every four bytes of metadata is +followed by a single byte of padding. + +This format is also used on CSI2 on 20 bits per sample format that packs two +bytes of metadata into one sample. + +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.** +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding. + +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}| + +.. flat-table:: + :header-rows: 0 + :stub-columns: 0 + :widths: 12 8 8 8 8 8 + + * - start + 0: + - M\ :sub:`00` + - M\ :sub:`10` + - M\ :sub:`20` + - M\ :sub:`30` + - p + * - start + 5: + - M\ :sub:`01` + - M\ :sub:`11` + - M\ :sub:`21` + - M\ :sub:`31` + - p + +.. _v4l2-meta-fmt-generic-csi2-12: + +V4L2_META_FMT_GENERIC_CSI2_12 +----------------------------- + +V4L2_META_FMT_GENERIC_CSI2_12 contains 8-bit generic metadata with CSI-2 +packing, 12 bits for each 8 bits of data. Every four bytes of metadata is +followed by two bytes of padding. + +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.** +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding. + +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}| + +.. flat-table:: + :header-rows: 0 + :stub-columns: 0 + :widths: 12 8 8 8 8 8 8 + + * - start + 0: + - M\ :sub:`00` + - M\ :sub:`10` + - M\ :sub:`20` + - M\ :sub:`30` + - p + - p + * - start + 6: + - M\ :sub:`01` + - M\ :sub:`11` + - M\ :sub:`21` + - M\ :sub:`31` + - p + - p + +.. _v4l2-meta-fmt-generic-csi2-14: + +V4L2_META_FMT_GENERIC_CSI2_14 +----------------------------- + +V4L2_META_FMT_GENERIC_CSI2_14 contains 8-bit generic metadata with CSI-2 +packing, 14 bits for each 8 bits of data. Every four bytes of metadata is +followed by three bytes of padding. + +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.** +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding. + +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}| + +.. flat-table:: + :header-rows: 0 + :stub-columns: 0 + :widths: 12 8 8 8 8 8 8 8 + + * - start + 0: + - M\ :sub:`00` + - M\ :sub:`10` + - M\ :sub:`20` + - M\ :sub:`30` + - p + - p + - p + * - start + 7: + - M\ :sub:`01` + - M\ :sub:`11` + - M\ :sub:`21` + - M\ :sub:`31` + - p + - p + - p + +.. _v4l2-meta-fmt-generic-csi2-16: + +V4L2_META_FMT_GENERIC_CSI2_16 +----------------------------- + +V4L2_META_FMT_GENERIC_CSI2_16 contains 8-bit generic metadata with CSI-2 +packing, 16 bits for each 8 bits of data. Every byte of metadata is +followed by one byte of padding. + +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.** +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding. + +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}| + +.. flat-table:: + :header-rows: 0 + :stub-columns: 0 + :widths: 12 8 8 8 8 8 8 8 8 + + * - start + 0: + - M\ :sub:`00` + - p + - M\ :sub:`10` + - p + - M\ :sub:`20` + - p + - M\ :sub:`30` + - p + * - start + 8: + - M\ :sub:`01` + - p + - M\ :sub:`11` + - p + - M\ :sub:`21` + - p + - M\ :sub:`31` + - p + +.. _v4l2-meta-fmt-generic-csi2-20: + +V4L2_META_FMT_GENERIC_CSI2_20 +----------------------------- + +V4L2_META_FMT_GENERIC_CSI2_20 contains 8-bit generic metadata with CSI-2 +packing, 20 bits for each 8 bits of data. Every byte of metadata is followed by +alternating one and two bytes of padding. + +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.** +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding. + +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}| + +.. flat-table:: + :header-rows: 0 + :stub-columns: 0 + :widths: 12 8 8 8 8 8 8 8 8 8 8 + + * - start + 0: + - M\ :sub:`00` + - p + - M\ :sub:`10` + - p + - p + - M\ :sub:`20` + - p + - M\ :sub:`30` + - p + - p + * - start + 10: + - M\ :sub:`01` + - p + - M\ :sub:`11` + - p + - p + - M\ :sub:`21` + - p + - M\ :sub:`31` + - p + - p + +.. _v4l2-meta-fmt-generic-csi2-24: + +V4L2_META_FMT_GENERIC_CSI2_24 +----------------------------- + +V4L2_META_FMT_GENERIC_CSI2_24 contains 8-bit generic metadata with CSI-2 +packing, 24 bits for each 8 bits of data. Every byte of metadata is followed by +alternating one and two bytes of padding. + +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.** +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding. + +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}| + +.. flat-table:: + :header-rows: 0 + :stub-columns: 0 + :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8 + + * - start + 0: + - M\ :sub:`00` + - p + - p + - M\ :sub:`10` + - p + - p + - M\ :sub:`20` + - p + - p + - M\ :sub:`30` + - p + - p + * - start + 12: + - M\ :sub:`01` + - p + - p + - M\ :sub:`11` + - p + - p + - M\ :sub:`21` + - p + - p + - M\ :sub:`31` + - p + - p + +.. _v4l2-meta-fmt-generic-csi2-2-24: + +V4L2_META_FMT_GENERIC_CSI2_2_24 +------------------------------- + +V4L2_META_FMT_GENERIC_CSI2_2_24 contains 8-bit generic metadata with CSI-2 +packing, 24 bits for each two times 8 bits of data. Every two bytes of metadata +are followed by one byte of padding. + +**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_2_24.** +Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding. + +.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{1.2cm}|p{.8cm}| + +.. flat-table:: + :header-rows: 0 + :stub-columns: 0 + :widths: 12 8 8 8 8 8 8 + + * - start + 0: + - M\ :sub:`00` + - M\ :sub:`10` + - p + - M\ :sub:`20` + - M\ :sub:`30` + - p + * - start + 6: + - M\ :sub:`01` + - M\ :sub:`11` + - p + - M\ :sub:`21` + - M\ :sub:`31` + - p +drivers/media/v4l2-core/v4l2-ioctl.c +8
-0@@ -1451,6 +1451,14 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)case V4L2_PIX_FMT_Y210 : descr ="10-bit YUYV Packed" ;break ;case V4L2_PIX_FMT_Y212 : descr ="12-bit YUYV Packed" ;break ;case V4L2_PIX_FMT_Y216 : descr ="16-bit YUYV Packed" ;break ; +case V4L2_META_FMT_GENERIC_8 : descr ="8-bit Generic Metadata" ;break ; +case V4L2_META_FMT_GENERIC_CSI2_10 : descr ="8b Generic Meta, 10b CSI-2" ;break ; +case V4L2_META_FMT_GENERIC_CSI2_12 : descr ="8b Generic Meta, 12b CSI-2" ;break ; +case V4L2_META_FMT_GENERIC_CSI2_14 : descr ="8b Generic Meta, 14b CSI-2" ;break ; +case V4L2_META_FMT_GENERIC_CSI2_16 : descr ="8b Generic Meta, 16b CSI-2" ;break ; +case V4L2_META_FMT_GENERIC_CSI2_20 : descr ="8b Generic Meta, 20b CSI-2" ;break ; +case V4L2_META_FMT_GENERIC_CSI2_24 : descr ="8b Generic Meta, 24b CSI-2" ;break ; +case V4L2_META_FMT_GENERIC_CSI2_2_24 : descr ="2x8b Generic Meta, 24b CSI-2" ;break ;default :/* Compressed formats */ include/uapi/linux/videodev2.h +9
-0@@ -834,6 +834,15 @@ struct v4l2_pix_format {#define V4L2_META_FMT_RK_ISP1_PARAMS v4l2_fourcc('R', 'K', '1', 'P')/* Rockchip ISP1 3A Parameters */ #define V4L2_META_FMT_RK_ISP1_STAT_3A v4l2_fourcc('R', 'K', '1', 'S')/* Rockchip ISP1 3A Statistics */ +#define V4L2_META_FMT_GENERIC_8 v4l2_fourcc('M', 'E', 'T', '8')/* Generic 8-bit metadata */ +#define V4L2_META_FMT_GENERIC_CSI2_10 v4l2_fourcc('M', 'C', '1', 'A')/* 10-bit CSI-2 packed 8-bit metadata */ +#define V4L2_META_FMT_GENERIC_CSI2_12 v4l2_fourcc('M', 'C', '1', 'C')/* 12-bit CSI-2 packed 8-bit metadata */ +#define V4L2_META_FMT_GENERIC_CSI2_14 v4l2_fourcc('M', 'C', '1', 'E')/* 14-bit CSI-2 packed 8-bit metadata */ +#define V4L2_META_FMT_GENERIC_CSI2_16 v4l2_fourcc('M', 'C', '1', 'G')/* 16-bit CSI-2 packed 8-bit metadata */ +#define V4L2_META_FMT_GENERIC_CSI2_20 v4l2_fourcc('M', 'C', '1', 'K')/* 20-bit CSI-2 packed 8-bit metadata */ +#define V4L2_META_FMT_GENERIC_CSI2_24 v4l2_fourcc('M', 'C', '1', 'O')/* 24-bit CSI-2 packed 8-bit metadata */ +#define V4L2_META_FMT_GENERIC_CSI2_2_24 v4l2_fourcc('M', 'C', '2', 'O')/* 2 bytes of 8-bit metadata, 24-bit CSI-2 packed */ +/* priv field value to indicates that subsequent fields are valid. */ #define V4L2_PIX_FMT_PRIV_MAGIC 0xfeedcafe
Sakari Ailus
maintainerRe: [RFC 6/7] media: uapi: Add generic 8-bit metadata format definitions
Hi Laurent,
Quoted 64 lines · Hans Verkuil
Quoted 50 lines · Sakari Ailus
Generic 8-bit metadata formats define the in-memory data layout but not the format of the data itself. The reasoning for having such formats is to allow CSI-2 receiver drivers to receive and DMA drivers to write the data to memory without knowing a large number of device specific formats.
These formats may be used only in conjunction of a Media controller pipeline where the internal pad of the source sub-device defines the specific format of the data (using an mbus code).
Signed-off-by Sakari Ailus <[email protected]>--- .../userspace-api/media/v4l/meta-formats.rst | 1 + .../media/v4l/metafmt-generic.rst | 317 ++++++++++++++++++ drivers/media/v4l2-core/v4l2-ioctl.c | 8 + include/uapi/linux/videodev2.h | 9 + 4 files changed, 335 insertions(+) create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rstDocumentation/userspace-api/media/v4l/meta-formats.rst +1
-0@@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only. metafmt-vsp1-hgo metafmt-vsp1-hgt metafmt-vivid + metafmt-genericDocumentation/userspace-api/media/v4l/metafmt-generic.rst new file +17
-0@@ -0,0 +1,317 @@ +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later + +************************************************************************************************************************************************************************************************************************************************************************************************************************** +V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O') +************************************************************************************************************************************************************************************************************************************************************************************************************************** + + +Generic line-based metadata formats + + +Description +=========== + +These generic line-based metadata formats define the memory layout of the data +without defining the format or meaning of the metadata itself. These formats may +only be used with a Media controller pipeline where the more specific format is +defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL-SOURCE>` of theDocumentation/userspace-api/media/v4l/metafmt-generic.rst line 27
Quoted 10 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +10
-0+source sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`. + +.. _v4l2-meta-fmt-generic-8: + +V4L2_META_FMT_GENERIC_8 +----------------------- + +The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format whereas the +rest of the formats have CSI-2 packing that are typically used on camera sensors +in conjunction with pixel data of the same bit depth.You keep referring to CSI-2 packing without ever defining it.
Quoted 4 lines
I'm not sure you should include CSI2 in the name either. It is OK to say something here like: "The way the data is stored follows the CSI-2 specification."
But it is not limited to CSI-2 and can be used with other buses as well.
That's true, certainly, but where this packing is most commonly used is CSI-2 (or derives from CSI-2).
Quoted 6 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst line 30
Quoted 4 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +3
-0+ +This format is also used on CSI-2 on both 8 bits per sample as well as on +16 bits per sample when two bytes of metadata are packed into one sample.Little or big endian?
Little endian. I'll address these for v2.
Signature · 3 lines
Kind regards, Sakari Ailus
Sakari Ailus
maintainerRFC 7/7 media: v4l: Support line-based metadata capture
In reply to Sakari Ailusmany camera sensors, among other devices, transmit embedded data and image data for each CSI-2 frame. This embedded data typically contains register configuration of the sensor that has been used to capture the image data of the same frame.
The embedded data is received by the CSI-2 receiver and has the same properties as the image data, including that it is line based: it has width, height and bytesperline (stride).
Add these fields to struct v4l2_meta_format and document them.
Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is line-based i.e. these fields of struct v4l2_meta_format are valid for it.
--- .../userspace-api/media/v4l/dev-meta.rst | 15 +++++++++++++++ .../userspace-api/media/v4l/vidioc-enum-fmt.rst | 7 +++++++ include/uapi/linux/videodev2.h | 10 ++++++++++ 3 files changed, 32 insertions(+)
Documentation/userspace-api/media/v4l/dev-meta.rst +15 -0
@@ -65,3 +65,18 @@ to 0. - ``buffersize`` - Maximum buffer size in bytes required for data. The value is set by the driver. + * - __u32 + - ``width`` + - Width of a line of metadata in bytes. Valid when :c:type`v4l2_fmtdesc` + flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See + :c:func:`VIDIOC_ENUM_FMT`. + * - __u32 + - ``height`` + - Height of a line of metadata in bytes. Valid when :c:type`v4l2_fmtdesc` + flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See + :c:func:`VIDIOC_ENUM_FMT`. + * - __u32 + - ``bytesperlines`` + - Offset in bytes between the beginning of two consecutive lines. Valid + when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is + set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +7 -0
@@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently: The application can ask to configure the quantization of the capture device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with :ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set. + * - ``V4L2_FMT_FLAG_META_LINE_BASED`` + - 0x0200 + - The metadata format is line-based. In this case the ``width``, + ``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are + valid. The buffer consists of ``height`` lines, each having ``width`` + bytes of data and offset between the beginning of each two consecutive + lines is ``bytesperline``. Return Value ============
include/uapi/linux/videodev2.h +10 -0
@@ -873,6 +873,7 @@ struct v4l2_fmtdesc {#define V4L2_FMT_FLAG_CSC_YCBCR_ENC 0x0080#define V4L2_FMT_FLAG_CSC_HSV_ENC V4L2_FMT_FLAG_CSC_YCBCR_ENC#define V4L2_FMT_FLAG_CSC_QUANTIZATION 0x0100 +#define V4L2_FMT_FLAG_META_LINE_BASED 0x0200/* Frame Size and frame rate enumeration */ /*
@@ -2407,10 +2408,19 @@ struct v4l2_sdr_format {* struct v4l2_meta_format - metadata format definition * @dataformat: little endian four character code (fourcc) * @buffersize: maximum size in bytes required for data +* @width: number of bytes of data per line (valid for line based +* formats only, see format documentation) +* @height: number of lines of data per buffer (valid for line based +* formats only) +* @bytesperline: offset between the beginnings of two adjacent lines in +* bytes (valid for line based formats only) */ struct v4l2_meta_format {__u32 dataformat;__u32 buffersize; +__u32 width; +__u32 height; +__u32 bytesperline; } __attribute__ ((packed));/**
Signature · 1 line
2.30.2
Laurent Pinchart
maintainerRe: [RFC 7/7] media: v4l: Support line-based metadata capture
Hi Sakari,
Thank you for the patch.
Quoted 1 line · Sakari Ailus
many camera sensors, among other devices, transmit embedded data and image
s/many/Many/
Quoted 25 lines
data for each CSI-2 frame. This embedded data typically contains register configuration of the sensor that has been used to capture the image data of the same frame.
The embedded data is received by the CSI-2 receiver and has the same properties as the image data, including that it is line based: it has width, height and bytesperline (stride).
Add these fields to struct
v4l2_meta_formatand document them.Also add
V4L2_FMT_FLAG_META_LINE_BASEDto tell a given format is line-based i.e. these fields of structv4l2_meta_formatare valid for it.Signed-off-by Sakari Ailus <[email protected]>--- .../userspace-api/media/v4l/dev-meta.rst | 15 +++++++++++++++ .../userspace-api/media/v4l/vidioc-enum-fmt.rst | 7 +++++++ include/uapi/linux/videodev2.h | 10 ++++++++++ 3 files changed, 32 insertions(+)Documentation/userspace-api/media/v4l/dev-meta.rst +0
-0
Documentation/userspace-api/media/v4l/dev-meta.rst to 0. · line 70
Quoted 6 lines
Documentation/userspace-api/media/v4l/dev-meta.rst +3
-0@@ -65,3 +65,18 @@ to 0. - ``buffersize`` - Maximum buffer size in bytes required for data. The value is set by the driver. + * - __u32 + - ``width`` + - Width of a line of metadata in bytes. Valid when :c:type`v4l2_fmtdesc`
This departs from pixel formats, where the width is defined in pixels. I wonder what the implications will be for userspace. Seeing one implementation, both in a kernel driver and in libcamera, will help validating the API.
Documentation/userspace-api/media/v4l/dev-meta.rst to 0. · line 75
Quoted 5 lines
Documentation/userspace-api/media/v4l/dev-meta.rst +5
-0+ flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See + :c:func:`VIDIOC_ENUM_FMT`. + * - __u32 + - ``height`` + - Height of a line of metadata in bytes. Valid when :c:type`v4l2_fmtdesc`
The "height of a line" seems like a weird concept, especially if the height is expressed in bytes. I assume this is a bad copy&paste.
Quoted 12 lines
Documentation/userspace-api/media/v4l/dev-meta.rst +7
-0+ flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See + :c:func:`VIDIOC_ENUM_FMT`. + * - __u32 + - ``bytesperlines`` + - Offset in bytes between the beginning of two consecutive lines. Valid + when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is + set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +0
-0
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst the ``mbus_code`` field is handled differently: · line 236
Quoted 10 lines
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +7
-0@@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently: The application can ask to configure the quantization of the capture device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with :ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set. + * - ``V4L2_FMT_FLAG_META_LINE_BASED`` + - 0x0200 + - The metadata format is line-based. In this case the ``width``, + ``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are + valid. The buffer consists of ``height`` lines, each having ``width`` + bytes of data and offset between the beginning of each two consecutive + lines is ``bytesperline``.
If we add width and height for metadata formats, does it mean that drivers have to (or can) implement VIDIOC_ENUM_FRAMESIZES ? This should be documented.
Quoted 35 lines
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +0
-0Return Value ============include/uapi/linux/videodev2.h +10
-0@@ -873,6 +873,7 @@ struct v4l2_fmtdesc {#define V4L2_FMT_FLAG_CSC_YCBCR_ENC 0x0080#define V4L2_FMT_FLAG_CSC_HSV_ENC V4L2_FMT_FLAG_CSC_YCBCR_ENC#define V4L2_FMT_FLAG_CSC_QUANTIZATION 0x0100 +#define V4L2_FMT_FLAG_META_LINE_BASED 0x0200/* Frame Size and frame rate enumeration */ /* @@ -2407,10 +2408,19 @@ struct v4l2_sdr_format {* struct v4l2_meta_format - metadata format definition * @dataformat: little endian four character code (fourcc) * @buffersize: maximum size in bytes required for data +* @width: number of bytes of data per line (valid for line based +* formats only, see format documentation) +* @height: number of lines of data per buffer (valid for line based +* formats only) +* @bytesperline: offset between the beginnings of two adjacent lines in +* bytes (valid for line based formats only) */ struct v4l2_meta_format {__u32 dataformat;__u32 buffersize; +__u32 width; +__u32 height; +__u32 bytesperline; } __attribute__ ((packed));/**
Signature · 3 lines
Regards, Laurent Pinchart
Sakari Ailus
maintainerRe: [RFC 7/7] media: v4l: Support line-based metadata capture
Hi Hans,
Quoted 29 lines · Laurent Pinchart
Hi Sakari,
Thank you for the patch.
Quoted 1 line · Sakari Ailus
many camera sensors, among other devices, transmit embedded data and image
s/many/Many/
Quoted 20 lines
data for each CSI-2 frame. This embedded data typically contains register configuration of the sensor that has been used to capture the image data of the same frame.
The embedded data is received by the CSI-2 receiver and has the same properties as the image data, including that it is line based: it has width, height and bytesperline (stride).
Add these fields to struct
v4l2_meta_formatand document them.Also add
V4L2_FMT_FLAG_META_LINE_BASEDto tell a given format is line-based i.e. these fields of structv4l2_meta_formatare valid for it.Signed-off-by Sakari Ailus <[email protected]>--- .../userspace-api/media/v4l/dev-meta.rst | 15 +++++++++++++++ .../userspace-api/media/v4l/vidioc-enum-fmt.rst | 7 +++++++ include/uapi/linux/videodev2.h | 10 ++++++++++ 3 files changed, 32 insertions(+)
Quoted 16 lines
Documentation/userspace-api/media/v4l/dev-meta.rst to 0. · line 70
Quoted 11 lines
Documentation/userspace-api/media/v4l/dev-meta.rst +3
-0@@ -65,3 +65,18 @@ to 0. - ``buffersize`` - Maximum buffer size in bytes required for data. The value is set by the driver. + * - __u32 + - ``width`` + - Width of a line of metadata in bytes. Valid when :c:type`v4l2_fmtdesc`This departs from pixel formats, where the width is defined in pixels. I wonder what the implications will be for userspace. Seeing one implementation, both in a kernel driver and in libcamera, will help validating the API.
I thought of bytes when writing this but pixels (or samples) are probably a better term for this.
Quoted 9 lines
Documentation/userspace-api/media/v4l/dev-meta.rst to 0. · line 75
Quoted 6 lines
Documentation/userspace-api/media/v4l/dev-meta.rst +5
-0+ flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See + :c:func:`VIDIOC_ENUM_FMT`. + * - __u32 + - ``height`` + - Height of a line of metadata in bytes. Valid when :c:type`v4l2_fmtdesc`The "height of a line" seems like a weird concept, especially if the height is expressed in bytes. I assume this is a bad copy&paste.
Yes. I'll address these for v2.
Quoted 19 lines
Quoted 19 lines
Documentation/userspace-api/media/v4l/dev-meta.rst +7
-0+ flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See + :c:func:`VIDIOC_ENUM_FMT`. + * - __u32 + - ``bytesperlines`` + - Offset in bytes between the beginning of two consecutive lines. Valid + when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is + set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +3
-0@@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently: The application can ask to configure the quantization of the capture device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with :ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set. + * - ``V4L2_FMT_FLAG_META_LINE_BASED`` + - 0x0200 + - The metadata format is line-based. In this case the ``width``,
Quoted 8 lines
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst the ``mbus_code`` field is handled differently: · line 236
Quoted 4 lines
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +4
-0+ ``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are + valid. The buffer consists of ``height`` lines, each having ``width`` + bytes of data and offset between the beginning of each two consecutive + lines is ``bytesperline``.If we add width and height for metadata formats, does it mean that drivers have to (or can) implement
VIDIOC_ENUM_FRAMESIZES? This should be documented.
Good point.
I don't think it's meaningful to implement that as the pipeline configuration determines this in any case.
But I wonder whether this would better be documented for V4L2_CAP_IO_MC rather than for metadata formats. There's currently only a reference to this in ENUM_FMT documentation.
Quoted 36 lines
Quoted 36 lines
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +0
-0Return Value ============include/uapi/linux/videodev2.h +10
-0@@ -873,6 +873,7 @@ struct v4l2_fmtdesc {#define V4L2_FMT_FLAG_CSC_YCBCR_ENC 0x0080#define V4L2_FMT_FLAG_CSC_HSV_ENC V4L2_FMT_FLAG_CSC_YCBCR_ENC#define V4L2_FMT_FLAG_CSC_QUANTIZATION 0x0100 +#define V4L2_FMT_FLAG_META_LINE_BASED 0x0200/* Frame Size and frame rate enumeration */ /* @@ -2407,10 +2408,19 @@ struct v4l2_sdr_format {* struct v4l2_meta_format - metadata format definition * @dataformat: little endian four character code (fourcc) * @buffersize: maximum size in bytes required for data +* @width: number of bytes of data per line (valid for line based +* formats only, see format documentation) +* @height: number of lines of data per buffer (valid for line based +* formats only) +* @bytesperline: offset between the beginnings of two adjacent lines in +* bytes (valid for line based formats only) */ struct v4l2_meta_format {__u32 dataformat;__u32 buffersize; +__u32 width; +__u32 height; +__u32 bytesperline; } __attribute__ ((packed));/**
Signature · 3 lines
Regards, Sakari Ailus
Hi Sakari,
Thank you for working on this. Sensor metadata is something that Raspberry Pi do make extensive use of, and our downstream changes to support it, although a bit hacky, are not too dissimilar to your proposal here.
Quoted 49 lines · Sakari Ailus
Hi folks,
Here are a few patches to add support generic, line based metadata as well as internal source pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.
Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.
Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc. The number of these formats is probably going to be about as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.
Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.
Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).
The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).
Quoted 4 lines
These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.
One thing that may be worth considering or clarifying - for the case of the BCM2835 Unicam CSI-2 device, we only have 2x DMA output channels. So one will match image data packets, and the other will match "everything else". Typically "everything else" would only be CSI-2 embedded data, but in the case of the Raspberry Pi Camera v3 (IMX708), it includes embedded data, PDAF data, and HDR histogram data. Each of these outputs can be programmed to use a different packet ID in the sensor, but since Unicam only has a single DMA for "everything else", it all gets dumped into one metadata buffer. But given we know the exact structure of the data streams, it's trivial for useland to find the right bits in this buffer. Of course, other CSI-2 receivers with more DMA channels might allow these streams to end up in their own buffers.
Nothing in your series seems to stop us operating Unicam in this way, particularly because there is no fixed definition of the data format for V4L2_META_FMT_GENERIC_8. So I don't think it's a problem, but perhaps it's worth documenting that the metadata might include multiple streams from the sensor?
Regards,
Naush
Quoted 57 lines
The internal (source) pads are an alternative to source routes [1]. The source routes were not universally liked and I do have to say I like re-using existing interface concepts (pads and everything you can do with pads, including access format, selections etc.) wherever it makes sense, instead of duplicating functionality.
Effectively internal source pads behave mostly just like sink pads, but they describe a flow of data that originates from a sub-device instead of arriving to a sub-device. The
SUBDEV_S_ROUTINGIOCTLs are used to enable and disable routes from internal source pads to sub-device's source pads. The subdev format IOCTLs are usable, too, so one can find which subdev format is available on given internal source pad.This set depends on these patches:
<URL:https://lore.kernel.org/linux-media/[email protected]/T/#t>
I've also pushed these here and I'll keep updating the branch:
<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>
Questions and comments are most welcome.
[1] <URL:https://lore.kernel.org/linux-media/[email protected]/>
Sakari Ailus (7):
media: mc: AddINTERNAL_SOURCEpad type flag
media: v4l: subdev: SupportINTERNAL_SOURCEpads in routing IOCTLs
media: uapi: v4l: Document source routes
media: mc: Check pad flag validity
media: uapi: Add generic serial metadata mbus formats
media: uapi: Add generic 8-bit metadata format definitions
media: v4l: Support line-based metadata capture.../media/mediactl/media-types.rst | 7 + .../userspace-api/media/v4l/dev-meta.rst | 15 + .../userspace-api/media/v4l/dev-subdev.rst | 18 + .../userspace-api/media/v4l/meta-formats.rst | 1 + .../media/v4l/metafmt-generic.rst | 317 ++++++++++++++++++ .../media/v4l/subdev-formats.rst | 257 ++++++++++++++ .../media/v4l/vidioc-enum-fmt.rst | 7 + .../media/v4l/vidioc-subdev-g-routing.rst | 5 + drivers/media/mc/mc-entity.c | 20 +- drivers/media/v4l2-core/v4l2-ioctl.c | 8 + drivers/media/v4l2-core/v4l2-subdev.c | 6 +- include/uapi/linux/media-bus-format.h | 9 + include/uapi/linux/media.h | 1 + include/uapi/linux/v4l2-subdev.h | 6 +- include/uapi/linux/videodev2.h | 19 ++ 15 files changed, 691 insertions(+), 5 deletions(-) create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst--
2.30.2
Sakari Ailus
maintainerHi Naushir,
Quoted 75 lines · Naushir Patuck
Hi Sakari,
Thank you for working on this. Sensor metadata is something that Raspberry Pi do make extensive use of, and our downstream changes to support it, although a bit hacky, are not too dissimilar to your proposal here.
Quoted 49 lines · Sakari Ailus
Hi folks,
Here are a few patches to add support generic, line based metadata as well as internal source pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.
Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.
Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc. The number of these formats is probably going to be about as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.
Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.
Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).
The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).Quoted 4 lines
These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.
One thing that may be worth considering or clarifying - for the case of the BCM2835 Unicam CSI-2 device, we only have 2x DMA output channels. So one will match image data packets, and the other will match "everything else". Typically "everything else" would only be CSI-2 embedded data, but in the case of the Raspberry Pi Camera v3 (IMX708), it includes embedded data, PDAF data, and HDR histogram data. Each of these outputs can be programmed to use a different packet ID in the sensor, but since Unicam only has a single DMA for "everything else", it all gets dumped into one metadata buffer. But given we know the exact structure of the data streams, it's trivial for useland to find the right bits in this buffer. Of course, other CSI-2 receivers with more DMA channels might allow these streams to end up in their own buffers.
Quoted 5 lines
Nothing in your series seems to stop us operating Unicam in this way, particularly because there is no fixed definition of the data format for
V4L2_META_FMT_GENERIC_8. So I don't think it's a problem, but perhaps it's worth documenting that the metadata might include multiple streams from the sensor?
I believe this happens on other hardware, too, indeed. Currently the documentation says that
Any number of routes from streams on sink pads towards streams on source pads is allowed, to the extent supported by drivers. For every stream on a source pad, however, only a single route is allowed.
(Documentation/userspace-api/media/v4l/dev-subdev.rst)
This probably needs to be changed to allow what you'd need?
Signature · 3 lines
Kind regards, Sakari Ailus
Hi Sakari,
Quoted 88 lines · Sakari Ailus
Hi Naushir,
Quoted 75 lines · Naushir Patuck
Hi Sakari,
Thank you for working on this. Sensor metadata is something that Raspberry Pi do make extensive use of, and our downstream changes to support it, although a bit hacky, are not too dissimilar to your proposal here.
Quoted 49 lines · Sakari Ailus
Hi folks,
Here are a few patches to add support generic, line based metadata as well as internal source pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.
Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.
Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc. The number of these formats is probably going to be about as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.
Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.
Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).
The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).Quoted 4 lines
These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.
One thing that may be worth considering or clarifying - for the case of the BCM2835 Unicam CSI-2 device, we only have 2x DMA output channels. So one will match image data packets, and the other will match "everything else". Typically "everything else" would only be CSI-2 embedded data, but in the case of the Raspberry Pi Camera v3 (IMX708), it includes embedded data, PDAF data, and HDR histogram data. Each of these outputs can be programmed to use a different packet ID in the sensor, but since Unicam only has a single DMA for "everything else", it all gets dumped into one metadata buffer. But given we know the exact structure of the data streams, it's trivial for useland to find the right bits in this buffer. Of course, other CSI-2 receivers with more DMA channels might allow these streams to end up in their own buffers.
Quoted 5 lines
Nothing in your series seems to stop us operating Unicam in this way, particularly because there is no fixed definition of the data format for
V4L2_META_FMT_GENERIC_8. So I don't think it's a problem, but perhaps it's worth documenting that the metadata might include multiple streams from the sensor?I believe this happens on other hardware, too, indeed. Currently the documentation says that
Quoted 8 lines
Any number of routes from streams on sink pads towards streams on source pads is allowed, to the extent supported by drivers. For every stream on a source pad, however, only a single route is allowed.
(
Documentation/userspace-api/media/v4l/dev-subdev.rst)This probably needs to be changed to allow what you'd need?
Yes, that last sentence sounds like it would (artificially wrt your series) limit
metadata buffers to only handle a single output stream. However, I may have got the context of the paragraph wrong as well :)
Regards,
Naush
Quoted 5 lines
--
Kind regards,Sakari Ailus
Sakari Ailus
maintainerHi Naush,
Quoted 96 lines · Naushir Patuck
Hi Sakari,
Quoted 93 lines · Sakari Ailus
Hi Naushir,
Quoted 75 lines · Naushir Patuck
Hi Sakari,
Thank you for working on this. Sensor metadata is something that Raspberry Pi do make extensive use of, and our downstream changes to support it, although a bit hacky, are not too dissimilar to your proposal here.
Quoted 49 lines · Sakari Ailus
Hi folks,
Here are a few patches to add support generic, line based metadata as well as internal source pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.
Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.
Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc. The number of these formats is probably going to be about as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.
Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.
Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).
The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).Quoted 4 lines
These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.
One thing that may be worth considering or clarifying - for the case of the BCM2835 Unicam CSI-2 device, we only have 2x DMA output channels. So one will match image data packets, and the other will match "everything else". Typically "everything else" would only be CSI-2 embedded data, but in the case of the Raspberry Pi Camera v3 (IMX708), it includes embedded data, PDAF data, and HDR histogram data. Each of these outputs can be programmed to use a different packet ID in the sensor, but since Unicam only has a single DMA for "everything else", it all gets dumped into one metadata buffer. But given we know the exact structure of the data streams, it's trivial for useland to find the right bits in this buffer. Of course, other CSI-2 receivers with more DMA channels might allow these streams to end up in their own buffers.
Quoted 5 lines
Nothing in your series seems to stop us operating Unicam in this way, particularly because there is no fixed definition of the data format for
V4L2_META_FMT_GENERIC_8. So I don't think it's a problem, but perhaps it's worth documenting that the metadata might include multiple streams from the sensor?I believe this happens on other hardware, too, indeed. Currently the documentation says that
Any number of routes from streams on sink pads towards streams on source pads is allowed, to the extent supported by drivers. For every stream on a source pad, however, only a single route is allowed.
Quoted 7 lines
Quoted 3 lines
(
Documentation/userspace-api/media/v4l/dev-subdev.rst)This probably needs to be changed to allow what you'd need?
Yes, that last sentence sounds like it would (artificially wrt your series) limit metadata buffers to only handle a single output stream. However, I may have got the context of the paragraph wrong as well :)
That was exactly the purpose: I wanted to make sure we didn't allow this without thinking what other implications it could have --- for instance what you also mentioned, the V4L2 format of the related buffer.
Signature · 3 lines
Kind regards, Sakari Ailus
Laurent Pinchart
maintainerHello,
Quoted 25 lines · Naushir Patuck
Hi Sakari,
Thank you for working on this. Sensor metadata is something that Raspberry Pi do make extensive use of, and our downstream changes to support it, although a bit hacky, are not too dissimilar to your proposal here.
Quoted 18 lines · Sakari Ailus
Hi folks,
Here are a few patches to add support generic, line based metadata as well as internal source pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.
Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.
Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc.
Optical dark pixels are image data, I wouldn't include them in the "metadata" category. They can of course be transmitted over a different stream, so they're relevant to the API being designed.
Quoted 5 lines
Quoted 5 lines
The number of these formats is probably going to be about
as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.
Strictly speaking, the number of metadata formats depends on how we define "format". Today, we can use the GREY pixel format to capture greyscale images in the visible spectrum, but also IR images, thermal images, or even depth images. They're all one pixel format. On the other hand, we have Y16 for greyscale visible and IR images, and Z16 for depth maps. It's already a mess, even without metadata :-)
Quoted 10 lines
Quoted 10 lines
Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.
I'm glad we agree on this.
Quoted 7 lines
Quoted 7 lines
Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).
Decoupling the information needed by the kernel (e.g. are we transporting RAW8 or RAW10 from the sensor through the pipeline) from the information useful for userspace only (e.g. the sensor embedded data is encoding using register + value pairs, based on the IMX708 registers set) is a good idea. I expect the main question to be where to draw the line between those two categories. Some pieces of information may be useless to any processing block in the pipeline except for an odd block in the middle.
This is, I believe, a problem similar to the CFA pattern. That information is useless for most devices, but required by the demosaicing block and some other blocks along the pipeline (colour gains for instance, or some Bayer statistics engines). We currently convey the CFA pattern in the media bus codes and pixel formats (e.g. SGRBG8 vs. SRGGB8) through the whole pipeline, while it could be conveyed out of band (e.g. exposed by the sensor using a control, and set on the devices that need it using a control as well).
If we come up with a good solution for metadata (and I hope we will), maybe we'll be able to use a similar mechanism for CFA patterns, simplifying new drivers and userspace. Or maybe this will remain a pipe dream given the backward compatibility implications.
Quoted 24 lines
Quoted 11 lines
The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.
One thing that may be worth considering or clarifying - for the case of the BCM2835 Unicam CSI-2 device, we only have 2x DMA output channels. So one will match image data packets, and the other will match "everything else". Typically "everything else" would only be CSI-2 embedded data, but in the case of the Raspberry Pi Camera v3 (IMX708), it includes embedded data, PDAF data, and HDR histogram data. Each of these outputs can be programmed to use a different packet ID in the sensor, but since Unicam only has a single DMA for "everything else", it all gets dumped into one metadata buffer. But given we know the exact structure of the data streams, it's trivial for useland to find the right bits in this buffer. Of course, other CSI-2 receivers with more DMA channels might allow these streams to end up in their own buffers.
Quoted 4 lines
Nothing in your series seems to stop us operating Unicam in this way, particularly because there is no fixed definition of the data format for
V4L2_META_FMT_GENERIC_8. So I don't think it's a problem, but perhaps it's worth documenting that the metadata might include multiple streams from the sensor?
Thanks for your feedback Naush. Would you consider reviewing the individual patches in the series ? :-)
Quoted 52 lines
Quoted 52 lines
The internal (source) pads are an alternative to source routes [1]. The source routes were not universally liked and I do have to say I like re-using existing interface concepts (pads and everything you can do with pads, including access format, selections etc.) wherever it makes sense, instead of duplicating functionality.
Effectively internal source pads behave mostly just like sink pads, but they describe a flow of data that originates from a sub-device instead of arriving to a sub-device. The
SUBDEV_S_ROUTINGIOCTLs are used to enable and disable routes from internal source pads to sub-device's source pads. The subdev format IOCTLs are usable, too, so one can find which subdev format is available on given internal source pad.This set depends on these patches:
<URL:https://lore.kernel.org/linux-media/[email protected]/T/#t>
I've also pushed these here and I'll keep updating the branch:
<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>
Questions and comments are most welcome.
[1] <URL:https://lore.kernel.org/linux-media/[email protected]/>
Sakari Ailus (7):
media: mc: AddINTERNAL_SOURCEpad type flag
media: v4l: subdev: SupportINTERNAL_SOURCEpads in routing IOCTLs
media: uapi: v4l: Document source routes
media: mc: Check pad flag validity
media: uapi: Add generic serial metadata mbus formats
media: uapi: Add generic 8-bit metadata format definitions
media: v4l: Support line-based metadata capture.../media/mediactl/media-types.rst | 7 + .../userspace-api/media/v4l/dev-meta.rst | 15 + .../userspace-api/media/v4l/dev-subdev.rst | 18 + .../userspace-api/media/v4l/meta-formats.rst | 1 + .../media/v4l/metafmt-generic.rst | 317 ++++++++++++++++++ .../media/v4l/subdev-formats.rst | 257 ++++++++++++++ .../media/v4l/vidioc-enum-fmt.rst | 7 + .../media/v4l/vidioc-subdev-g-routing.rst | 5 + drivers/media/mc/mc-entity.c | 20 +- drivers/media/v4l2-core/v4l2-ioctl.c | 8 + drivers/media/v4l2-core/v4l2-subdev.c | 6 +- include/uapi/linux/media-bus-format.h | 9 + include/uapi/linux/media.h | 1 + include/uapi/linux/v4l2-subdev.h | 6 +- include/uapi/linux/videodev2.h | 19 ++ 15 files changed, 691 insertions(+), 5 deletions(-) create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
Signature · 3 lines
Regards, Laurent Pinchart
Quoted 115 lines · Laurent Pinchart
Hello,
Quoted 25 lines · Naushir Patuck
Hi Sakari,
Thank you for working on this. Sensor metadata is something that Raspberry Pi do make extensive use of, and our downstream changes to support it, although a bit hacky, are not too dissimilar to your proposal here.
Quoted 18 lines · Sakari Ailus
Hi folks,
Here are a few patches to add support generic, line based metadata as well as internal source pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.
Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.
Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc.
Optical dark pixels are image data, I wouldn't include them in the "metadata" category. They can of course be transmitted over a different stream, so they're relevant to the API being designed.
Quoted 5 lines
Quoted 5 lines
The number of these formats is probably going to be about
as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.Strictly speaking, the number of metadata formats depends on how we define "format". Today, we can use the GREY pixel format to capture greyscale images in the visible spectrum, but also IR images, thermal images, or even depth images. They're all one pixel format. On the other hand, we have Y16 for greyscale visible and IR images, and Z16 for depth maps. It's already a mess, even without metadata :-)
Quoted 10 lines
Quoted 10 lines
Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.
I'm glad we agree on this.
Quoted 7 lines
Quoted 7 lines
Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).
Decoupling the information needed by the kernel (e.g. are we transporting RAW8 or RAW10 from the sensor through the pipeline) from the information useful for userspace only (e.g. the sensor embedded data is encoding using register + value pairs, based on the IMX708 registers set) is a good idea. I expect the main question to be where to draw the line between those two categories. Some pieces of information may be useless to any processing block in the pipeline except for an odd block in the middle.
This is, I believe, a problem similar to the CFA pattern. That information is useless for most devices, but required by the demosaicing block and some other blocks along the pipeline (colour gains for instance, or some Bayer statistics engines). We currently convey the CFA pattern in the media bus codes and pixel formats (e.g. SGRBG8 vs. SRGGB8) through the whole pipeline, while it could be conveyed out of band (e.g. exposed by the sensor using a control, and set on the devices that need it using a control as well).
If we come up with a good solution for metadata (and I hope we will), maybe we'll be able to use a similar mechanism for CFA patterns, simplifying new drivers and userspace. Or maybe this will remain a pipe dream given the backward compatibility implications.
Quoted 24 lines
Quoted 11 lines
The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.
One thing that may be worth considering or clarifying - for the case of the BCM2835 Unicam CSI-2 device, we only have 2x DMA output channels. So one will match image data packets, and the other will match "everything else". Typically "everything else" would only be CSI-2 embedded data, but in the case of the Raspberry Pi Camera v3 (IMX708), it includes embedded data, PDAF data, and HDR histogram data. Each of these outputs can be programmed to use a different packet ID in the sensor, but since Unicam only has a single DMA for "everything else", it all gets dumped into one metadata buffer. But given we know the exact structure of the data streams, it's trivial for useland to find the right bits in this buffer. Of course, other CSI-2 receivers with more DMA channels might allow these streams to end up in their own buffers.
Quoted 7 lines
Quoted 4 lines
Nothing in your series seems to stop us operating Unicam in this way, particularly because there is no fixed definition of the data format for
V4L2_META_FMT_GENERIC_8. So I don't think it's a problem, but perhaps it's worth documenting that the metadata might include multiple streams from the sensor?Thanks for your feedback Naush. Would you consider reviewing the individual patches in the series ? :-)
Sure, I'll go through the patches and send some feedback. This is after all a change that I'm very much looking forward to :-)
Regards,
Naush
Quoted 58 lines
Quoted 45 lines
Quoted 45 lines
The internal (source) pads are an alternative to source routes [1]. The source routes were not universally liked and I do have to say I like re-using existing interface concepts (pads and everything you can do with pads, including access format, selections etc.) wherever it makes sense, instead of duplicating functionality.
Effectively internal source pads behave mostly just like sink pads, but they describe a flow of data that originates from a sub-device instead of arriving to a sub-device. The
SUBDEV_S_ROUTINGIOCTLs are used to enable and disable routes from internal source pads to sub-device's source pads. The subdev format IOCTLs are usable, too, so one can find which subdev format is available on given internal source pad.This set depends on these patches:
<URL:https://lore.kernel.org/linux-media/[email protected]/T/#t>
I've also pushed these here and I'll keep updating the branch:
<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>
Questions and comments are most welcome.
[1] <URL:https://lore.kernel.org/linux-media/[email protected]/>
Sakari Ailus (7):
media: mc: AddINTERNAL_SOURCEpad type flag
media: v4l: subdev: SupportINTERNAL_SOURCEpads in routing IOCTLs
media: uapi: v4l: Document source routes
media: mc: Check pad flag validity
media: uapi: Add generic serial metadata mbus formats
media: uapi: Add generic 8-bit metadata format definitions
media: v4l: Support line-based metadata capture.../media/mediactl/media-types.rst | 7 + .../userspace-api/media/v4l/dev-meta.rst | 15 + .../userspace-api/media/v4l/dev-subdev.rst | 18 + .../userspace-api/media/v4l/meta-formats.rst | 1 + .../media/v4l/metafmt-generic.rst | 317 ++++++++++++++++++ .../media/v4l/subdev-formats.rst | 257 ++++++++++++++ .../media/v4l/vidioc-enum-fmt.rst | 7 + .../media/v4l/vidioc-subdev-g-routing.rst | 5 + drivers/media/mc/mc-entity.c | 20 +-Quoted 8 lines
Quoted 8 lines
drivers/media/v4l2-core/v4l2-ioctl.c | 8 + drivers/media/v4l2-core/v4l2-subdev.c | 6 +- include/uapi/linux/media-bus-format.h | 9 + include/uapi/linux/media.h | 1 + include/uapi/linux/v4l2-subdev.h | 6 +- include/uapi/linux/videodev2.h | 19 ++ 15 files changed, 691 insertions(+), 5 deletions(-) create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst--
Regards,Laurent Pinchart
Sakari Ailus
maintainerHi Laurent,
Quoted 24 lines · Laurent Pinchart
Hello,
Quoted 21 lines · Naushir Patuck
Hi Sakari,
Thank you for working on this. Sensor metadata is something that Raspberry Pi do make extensive use of, and our downstream changes to support it, although a bit hacky, are not too dissimilar to your proposal here.
Quoted 14 lines · Sakari Ailus
Hi folks,
Here are a few patches to add support generic, line based metadata as well as internal source pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.
Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc.
Optical dark pixels are image data, I wouldn't include them in the "metadata" category. They can of course be transmitted over a different stream, so they're relevant to the API being designed.
Good point, I'll address this for v2.
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
The number of these formats is probably going to be about
as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.
Quoted 6 lines
Strictly speaking, the number of metadata formats depends on how we define "format". Today, we can use the GREY pixel format to capture greyscale images in the visible spectrum, but also IR images, thermal images, or even depth images. They're all one pixel format. On the other hand, we have Y16 for greyscale visible and IR images, and Z16 for depth maps. It's already a mess, even without metadata :-)
Yes.
Quoted 12 lines
Quoted 11 lines
Quoted 11 lines
Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.
Quoted 1 line
I'm glad we agree on this.
Well, there are alternatives, but this is probably easier in the long run.
We'll just need more documentation than usually for the metadata mbus codes.
Quoted 18 lines
Quoted 8 lines
Quoted 8 lines
Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).
Decoupling the information needed by the kernel (e.g. are we transporting RAW8 or RAW10 from the sensor through the pipeline) from the information useful for userspace only (e.g. the sensor embedded data is encoding using register + value pairs, based on the IMX708 registers set) is a good idea. I expect the main question to be where to draw the line between those two categories. Some pieces of information may be useless to any processing block in the pipeline except for an odd block in the middle.
Quoted 8 lines
This is, I believe, a problem similar to the CFA pattern. That information is useless for most devices, but required by the demosaicing block and some other blocks along the pipeline (colour gains for instance, or some Bayer statistics engines). We currently convey the CFA pattern in the media bus codes and pixel formats (e.g. SGRBG8 vs. SRGGB8) through the whole pipeline, while it could be conveyed out of band (e.g. exposed by the sensor using a control, and set on the devices that need it using a control as well).
Cases where another device in the pipeline would need to know the exact metadata format are exceedingly rare. I actually don't have any I could name right now.
Quoted 90 lines
If we come up with a good solution for metadata (and I hope we will), maybe we'll be able to use a similar mechanism for CFA patterns, simplifying new drivers and userspace. Or maybe this will remain a pipe dream given the backward compatibility implications.
Quoted 24 lines
Quoted 11 lines
The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.
One thing that may be worth considering or clarifying - for the case of the BCM2835 Unicam CSI-2 device, we only have 2x DMA output channels. So one will match image data packets, and the other will match "everything else". Typically "everything else" would only be CSI-2 embedded data, but in the case of the Raspberry Pi Camera v3 (IMX708), it includes embedded data, PDAF data, and HDR histogram data. Each of these outputs can be programmed to use a different packet ID in the sensor, but since Unicam only has a single DMA for "everything else", it all gets dumped into one metadata buffer. But given we know the exact structure of the data streams, it's trivial for useland to find the right bits in this buffer. Of course, other CSI-2 receivers with more DMA channels might allow these streams to end up in their own buffers.
Quoted 4 lines
Nothing in your series seems to stop us operating Unicam in this way, particularly because there is no fixed definition of the data format for
V4L2_META_FMT_GENERIC_8. So I don't think it's a problem, but perhaps it's worth documenting that the metadata might include multiple streams from the sensor?Thanks for your feedback Naush. Would you consider reviewing the individual patches in the series ? :-)
Quoted 52 lines
Quoted 52 lines
The internal (source) pads are an alternative to source routes [1]. The source routes were not universally liked and I do have to say I like re-using existing interface concepts (pads and everything you can do with pads, including access format, selections etc.) wherever it makes sense, instead of duplicating functionality.
Effectively internal source pads behave mostly just like sink pads, but they describe a flow of data that originates from a sub-device instead of arriving to a sub-device. The
SUBDEV_S_ROUTINGIOCTLs are used to enable and disable routes from internal source pads to sub-device's source pads. The subdev format IOCTLs are usable, too, so one can find which subdev format is available on given internal source pad.This set depends on these patches:
<URL:https://lore.kernel.org/linux-media/[email protected]/T/#t>
I've also pushed these here and I'll keep updating the branch:
<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>
Questions and comments are most welcome.
[1] <URL:https://lore.kernel.org/linux-media/[email protected]/>
Sakari Ailus (7):
media: mc: AddINTERNAL_SOURCEpad type flag
media: v4l: subdev: SupportINTERNAL_SOURCEpads in routing IOCTLs
media: uapi: v4l: Document source routes
media: mc: Check pad flag validity
media: uapi: Add generic serial metadata mbus formats
media: uapi: Add generic 8-bit metadata format definitions
media: v4l: Support line-based metadata capture.../media/mediactl/media-types.rst | 7 + .../userspace-api/media/v4l/dev-meta.rst | 15 + .../userspace-api/media/v4l/dev-subdev.rst | 18 + .../userspace-api/media/v4l/meta-formats.rst | 1 + .../media/v4l/metafmt-generic.rst | 317 ++++++++++++++++++ .../media/v4l/subdev-formats.rst | 257 ++++++++++++++ .../media/v4l/vidioc-enum-fmt.rst | 7 + .../media/v4l/vidioc-subdev-g-routing.rst | 5 + drivers/media/mc/mc-entity.c | 20 +- drivers/media/v4l2-core/v4l2-ioctl.c | 8 + drivers/media/v4l2-core/v4l2-subdev.c | 6 +- include/uapi/linux/media-bus-format.h | 9 + include/uapi/linux/media.h | 1 + include/uapi/linux/v4l2-subdev.h | 6 +- include/uapi/linux/videodev2.h | 19 ++ 15 files changed, 691 insertions(+), 5 deletions(-) create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
Signature · 3 lines
Kind regards, Sakari Ailus
Hi Sakari
Sorry to be late to the party.
Quoted 60 lines · Sakari Ailus
Hi folks,
Here are a few patches to add support generic, line based metadata as well as internal source pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.
Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.
Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc. The number of these formats is probably going to be about as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.
Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.
Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).
The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.
The internal (source) pads are an alternative to source routes [1]. The source routes were not universally liked and I do have to say I like re-using existing interface concepts (pads and everything you can do with pads, including access format, selections etc.) wherever it makes sense, instead of duplicating functionality.
Quoted 6 lines
Effectively internal source pads behave mostly just like sink pads, but they describe a flow of data that originates from a sub-device instead of arriving to a sub-device. The
SUBDEV_S_ROUTINGIOCTLs are used to enable and disable routes from internal source pads to sub-device's source pads. The subdev format IOCTLs are usable, too, so one can find which subdev format is available on given internal source pad.
Just to confirm my own understanding, the muxed streams API currently copes with situations such as the FPD-Link devices combining a single stream from each of a number of subdevs into one multiplexed stream using virtual channels, but it doesn't handle multiple data types from the same subdev. You're addressing that omission.
All seems reasonable.
One detail that I haven't seen covered and think ought to be clarified. For raw sensors where you have image data and metadata, in my experience the line width of that metadata and the packing format (8/10/12/14/16/20/24 bit) are all dictated by the image data configuration. You can't have eg 640 pixel wide _SBGGR10_1X10 image data and 320 pixel wide _META_1X8_8 metadata. (it has to be 640 pixel wide _META_1X10_10)
Which layer would be responsible for validating the configuration, and when?
a) The driver validates in set_stream, and fails if mismatched. That potentially results in lots of duplication between drivers.
b) set_pad_format on the image pad updates the formats and widths of the metadata pads automatically. Is there a way of notifying clients that the formats on potentially unrelated pads has changed, or is it something that just gets documented?
c) something else.
Either is workable, but IMHO it needs to be specified as to which is the expected behaviour.
Thanks
Dave
Quoted 43 lines
This set depends on these patches:
<URL:https://lore.kernel.org/linux-media/[email protected]/T/#t>
I've also pushed these here and I'll keep updating the branch:
<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>
Questions and comments are most welcome.
[1] <URL:https://lore.kernel.org/linux-media/[email protected]/>
Sakari Ailus (7):
media: mc: AddINTERNAL_SOURCEpad type flag
media: v4l: subdev: SupportINTERNAL_SOURCEpads in routing IOCTLs
media: uapi: v4l: Document source routes
media: mc: Check pad flag validity
media: uapi: Add generic serial metadata mbus formats
media: uapi: Add generic 8-bit metadata format definitions
media: v4l: Support line-based metadata capture.../media/mediactl/media-types.rst | 7 + .../userspace-api/media/v4l/dev-meta.rst | 15 + .../userspace-api/media/v4l/dev-subdev.rst | 18 + .../userspace-api/media/v4l/meta-formats.rst | 1 + .../media/v4l/metafmt-generic.rst | 317 ++++++++++++++++++ .../media/v4l/subdev-formats.rst | 257 ++++++++++++++ .../media/v4l/vidioc-enum-fmt.rst | 7 + .../media/v4l/vidioc-subdev-g-routing.rst | 5 + drivers/media/mc/mc-entity.c | 20 +- drivers/media/v4l2-core/v4l2-ioctl.c | 8 + drivers/media/v4l2-core/v4l2-subdev.c | 6 +- include/uapi/linux/media-bus-format.h | 9 + include/uapi/linux/media.h | 1 + include/uapi/linux/v4l2-subdev.h | 6 +- include/uapi/linux/videodev2.h | 19 ++ 15 files changed, 691 insertions(+), 5 deletions(-) create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst--
2.30.2
Sakari Ailus
maintainerHi Dave,
Quoted 72 lines · Dave Stevenson
Hi Sakari
Sorry to be late to the party.
Quoted 60 lines · Sakari Ailus
Hi folks,
Here are a few patches to add support generic, line based metadata as well as internal source pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.
Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.
Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc. The number of these formats is probably going to be about as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.
Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.
Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).
The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.
The internal (source) pads are an alternative to source routes [1]. The source routes were not universally liked and I do have to say I like re-using existing interface concepts (pads and everything you can do with pads, including access format, selections etc.) wherever it makes sense, instead of duplicating functionality.
Quoted 6 lines
Effectively internal source pads behave mostly just like sink pads, but they describe a flow of data that originates from a sub-device instead of arriving to a sub-device. The
SUBDEV_S_ROUTINGIOCTLs are used to enable and disable routes from internal source pads to sub-device's source pads. The subdev format IOCTLs are usable, too, so one can find which subdev format is available on given internal source pad.
Quoted 5 lines
Just to confirm my own understanding, the muxed streams API currently copes with situations such as the FPD-Link devices combining a single stream from each of a number of subdevs into one multiplexed stream using virtual channels, but it doesn't handle multiple data types from the same subdev. You're addressing that omission.
Correct.
Quoted 10 lines
All seems reasonable.
One detail that I haven't seen covered and think ought to be clarified. For raw sensors where you have image data and metadata, in my experience the line width of that metadata and the packing format (8/10/12/14/16/20/24 bit) are all dictated by the image data configuration. You can't have eg 640 pixel wide
_SBGGR10_1X10image data and 320 pixel wide_META_1X8_8metadata. (it has to be 640 pixel wide_META_1X10_10)
That corresponds to my experience and is probably related to sensor implementation rather than CSI-2 itself.
Quoted 10 lines
Which layer would be responsible for validating the configuration, and when?
a) The driver validates inset_stream, and fails if mismatched. That potentially results in lots of duplication between drivers.
b)set_pad_formaton the image pad updates the formats and widths of the metadata pads automatically. Is there a way of notifying clients that the formats on potentially unrelated pads has changed, or is it something that just gets documented?
c) something else.
Quoted 2 lines
Either is workable, but IMHO it needs to be specified as to which is the expected behaviour.
I left that out of the spec as in the end this depends on the hardware constraints and how the driver author decided to map that to the API.
I expect sensor drivers to let the user configure the image format freely (as it's been so far) and if there's anything to configure in the metadata format that does not affect the image data format, then that configurability is available on the metadata internal sink pad via SUBDEV_S_FMT. So from the above options b appears a good fit. SUBDEV_S_FMT does not have notification ability and probably shouldn't, instead the order of configuration should be followed.
I think this would be good to be documented, perhaps with the first sensor driver that uses it. Likely to be merged with this set later on.
Signature · 3 lines
Kind regards, Sakari Ailus
Laurent Pinchart
maintainerQuoted 87 lines · Sakari Ailus
Quoted 68 lines · Dave Stevenson
Quoted 60 lines · Sakari Ailus
Hi folks,
Here are a few patches to add support generic, line based metadata as well as internal source pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.
Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.
Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc. The number of these formats is probably going to be about as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.
Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.
Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).
The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.
The internal (source) pads are an alternative to source routes [1]. The source routes were not universally liked and I do have to say I like re-using existing interface concepts (pads and everything you can do with pads, including access format, selections etc.) wherever it makes sense, instead of duplicating functionality.
Quoted 6 lines
Effectively internal source pads behave mostly just like sink pads, but they describe a flow of data that originates from a sub-device instead of arriving to a sub-device. The
SUBDEV_S_ROUTINGIOCTLs are used to enable and disable routes from internal source pads to sub-device's source pads. The subdev format IOCTLs are usable, too, so one can find which subdev format is available on given internal source pad.Quoted 5 lines
Just to confirm my own understanding, the muxed streams API currently copes with situations such as the FPD-Link devices combining a single stream from each of a number of subdevs into one multiplexed stream using virtual channels, but it doesn't handle multiple data types from the same subdev. You're addressing that omission.
Correct.
Quoted 9 lines
All seems reasonable.
One detail that I haven't seen covered and think ought to be clarified. For raw sensors where you have image data and metadata, in my experience the line width of that metadata and the packing format (8/10/12/14/16/20/24 bit) are all dictated by the image data configuration. You can't have eg 640 pixel wide
_SBGGR10_1X10image data and 320 pixel wide_META_1X8_8metadata. (it has to be 640 pixel wide_META_1X10_10)
Quoted 2 lines
That corresponds to my experience and is probably related to sensor implementation rather than CSI-2 itself.
I agree, in theory the above could be implemented by a CSI-2 source, but in practice that would be very uncommon.
Quoted 23 lines
Quoted 11 lines
Which layer would be responsible for validating the configuration, and when?
a) The driver validates inset_stream, and fails if mismatched. That potentially results in lots of duplication between drivers.
b)set_pad_formaton the image pad updates the formats and widths of the metadata pads automatically. Is there a way of notifying clients that the formats on potentially unrelated pads has changed, or is it something that just gets documented?
c) something else.Either is workable, but IMHO it needs to be specified as to which is the expected behaviour.
I left that out of the spec as in the end this depends on the hardware constraints and how the driver author decided to map that to the API.
I expect sensor drivers to let the user configure the image format freely (as it's been so far) and if there's anything to configure in the metadata format that does not affect the image data format, then that configurability is available on the metadata internal sink pad via
SUBDEV_S_FMT. So from the above options b appears a good fit.SUBDEV_S_FMTdoes not have notification ability and probably shouldn't, instead the order of configuration should be followed.
Quoted 2 lines
I think this would be good to be documented, perhaps with the first sensor driver that uses it. Likely to be merged with this set later on.
I agree with the above. The behaviour depends on the hardware constraints, and is thus device-dependent, but we can standardize it for classes of devices. For raw sensors that produce image and embedded data streams, option B should be required in the documentation.
Signature · 3 lines
Regards, Laurent Pinchart