Skip to contentSkip to conversation

Generic line based metadata support, internal pads

RFC Superseded Sakari Ailus 6 participants 55 messages May 5, 2023 – Aug 3, 2023

Subsystems (2)

v1 of 9 Superseded Latest is v9 →

lore
Share & tools
lore

Sakari Ailus

author maintainer
#0 lore raw

cover Generic line based metadata support, internal pads

Hi folks,

Here are a few patches to add support generic, line based metadata as well as internal source pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.

Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.

Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc. The number of these formats is probably going to be about as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.

Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.

Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).

The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).

These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.

The internal (source) pads are an alternative to source routes [1]. The source routes were not universally liked and I do have to say I like re-using existing interface concepts (pads and everything you can do with pads, including access format, selections etc.) wherever it makes sense, instead of duplicating functionality.

Effectively internal source pads behave mostly just like sink pads, but they describe a flow of data that originates from a sub-device instead of arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable and disable routes from internal source pads to sub-device's source pads. The subdev format IOCTLs are usable, too, so one can find which subdev format is available on given internal source pad.

This set depends on these patches:

<URL:https://lore.kernel.org/linux-media/[email protected]/T/#t>

I've also pushed these here and I'll keep updating the branch:

<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>

Questions and comments are most welcome.

[1] <URL:https://lore.kernel.org/linux-media/[email protected]/>

Sakari Ailus (7):
media: mc: Add INTERNAL_SOURCE pad type flag
media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs
media: uapi: v4l: Document source routes
media: mc: Check pad flag validity
media: uapi: Add generic serial metadata mbus formats
media: uapi: Add generic 8-bit metadata format definitions
media: v4l: Support line-based metadata capture

 .../media/mediactl/media-types.rst            |   7 +
 .../userspace-api/media/v4l/dev-meta.rst      |  15 +
 .../userspace-api/media/v4l/dev-subdev.rst    |  18 +
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 317 ++++++++++++++++++
 .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++
 .../media/v4l/vidioc-enum-fmt.rst             |   7 +
 .../media/v4l/vidioc-subdev-g-routing.rst     |   5 +
 drivers/media/mc/mc-entity.c                  |  20 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
 drivers/media/v4l2-core/v4l2-subdev.c         |   6 +-
 include/uapi/linux/media-bus-format.h         |   9 +
 include/uapi/linux/media.h                    |   1 +
 include/uapi/linux/v4l2-subdev.h              |   6 +-
 include/uapi/linux/videodev2.h                |  19 ++
 15 files changed, 691 insertions(+), 5 deletions(-)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
Signature · 1 line
2.30.2
author maintainer
#1 lore raw

RFC 1/7 media: mc: Add INTERNAL_SOURCE pad type flag

Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.

Also prevent creating links to pads that have been flagged as internal.

---
 .../userspace-api/media/mediactl/media-types.rst          | 7 +++++++
 drivers/media/mc/mc-entity.c                              | 8 +++++++-
 include/uapi/linux/media.h                                | 1 +
 3 files changed, 15 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/mediactl/media-types.rst +7 -0
index 0ffeece1e0c8..c724139ad46c 100644
@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
 .. _MEDIA-PAD-FL-SINK:
 .. _MEDIA-PAD-FL-SOURCE:
 .. _MEDIA-PAD-FL-MUST-CONNECT:
+.. _MEDIA-PAD-FL-INTERNAL-SOURCE:
 
 .. flat-table:: Media pad flags
     :header-rows:  0
@@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements
 	  when this flag isn't set; the absence of the flag doesn't imply
 	  there is none.
 
+    *  -  ``MEDIA_PAD_FL_INTERNAL_SOURCE``
+       -  This flag indicates an internal pad that has no external
+	  connections. Such a pad may not be connected with a link. The internal
+	  flag indicates that the stream either starts or ends in the
+	  entity. For a given pad, the INTERNAL_SOURCE flag may not be set if
+	  either SINK or SOURCE flags is set.
 
 One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
 must be set for every pad.
drivers/media/mc/mc-entity.c +7 -1
index ef34ddd715c9..ed99160a2487 100644
@@ -1062,7 +1062,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
 
 	for (i = 0; i < entity->num_pads; i++) {
 		if ((entity->pads[i].flags &
-		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
+		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
+		      MEDIA_PAD_FL_INTERNAL_SOURCE)) != pad_type)
 			continue;
 
 		if (entity->pads[i].sig_type == sig_type)
@@ -1087,6 +1088,11 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
 		return -EINVAL;
 	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
 		return -EINVAL;
+	if (WARN_ON(sink->pads[sink_pad].flags &
+		    MEDIA_PAD_FL_INTERNAL_SOURCE) ||
+	    WARN_ON(source->pads[source_pad].flags &
+		    MEDIA_PAD_FL_INTERNAL_SOURCE))
+		return -EINVAL;
 
 	link = media_add_link(&source->links);
 	if (link == NULL)
include/uapi/linux/media.h +1 -0
index edb8dfef9eba..0e2577e8b425 100644
@@ -208,6 +208,7 @@ struct media_entity_desc {
 #define MEDIA_PAD_FL_SINK			(1U << 0)
 #define MEDIA_PAD_FL_SOURCE			(1U << 1)
 #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
+#define MEDIA_PAD_FL_INTERNAL_SOURCE		(1U << 3)
 
 struct media_pad_desc {
 	__u32 entity;		/* entity ID */
Signature · 1 line
2.30.2
#2 lore raw

Re: [RFC 1/7] media: mc: Add INTERNAL_SOURCE pad type flag

Quoted 25 lines · Sakari Ailus

Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.

Also prevent creating links to pads that have been flagged as internal.

---
  .../userspace-api/media/mediactl/media-types.rst          | 7 +++++++
  drivers/media/mc/mc-entity.c                              | 8 +++++++-
  include/uapi/linux/media.h                                | 1 +
  3 files changed, 15 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/mediactl/media-types.rst +1 -0
index 0ffeece1e0c8..c724139ad46c 100644
@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
  .. _MEDIA-PAD-FL-SINK:
  .. _MEDIA-PAD-FL-SOURCE:
  .. _MEDIA-PAD-FL-MUST-CONNECT:
+.. _MEDIA-PAD-FL-INTERNAL-SOURCE:
 
  .. flat-table:: Media pad flags
      :header-rows:  0

Documentation/userspace-api/media/mediactl/media-types.rst Types and flags used to represent the media graph elements · line 388

Quoted 6 lines
Documentation/userspace-api/media/mediactl/media-types.rst +3 -0
@@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements
  	  when this flag isn't set; the absence of the flag doesn't imply
  	  there is none.
 
+    *  -  ``MEDIA_PAD_FL_INTERNAL_SOURCE``
+       -  This flag indicates an internal pad that has no external
+	  connections. Such a pad may not be connected with a link. The internal

"must not"? Or "shall not"?

Tomi

author maintainer
#3 lore raw

Re: [RFC 1/7] media: mc: Add INTERNAL_SOURCE pad type flag

Hi Tomi,

Thank you for the review.

Quoted 24 lines · Tomi Valkeinen
Quoted 23 lines · Sakari Ailus

Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.

Also prevent creating links to pads that have been flagged as internal.

---
  .../userspace-api/media/mediactl/media-types.rst          | 7 +++++++
  drivers/media/mc/mc-entity.c                              | 8 +++++++-
  include/uapi/linux/media.h                                | 1 +
  3 files changed, 15 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/mediactl/media-types.rst +1 -0
index 0ffeece1e0c8..c724139ad46c 100644
@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
  .. _MEDIA-PAD-FL-SINK:
  .. _MEDIA-PAD-FL-SOURCE:
  .. _MEDIA-PAD-FL-MUST-CONNECT:
+.. _MEDIA-PAD-FL-INTERNAL-SOURCE:
  .. flat-table:: Media pad flags
      :header-rows:  0
Quoted 8 lines

Documentation/userspace-api/media/mediactl/media-types.rst Types and flags used to represent the media graph elements · line 387

Quoted 6 lines
Documentation/userspace-api/media/mediactl/media-types.rst +3 -0
@@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements
  	  when this flag isn't set; the absence of the flag doesn't imply
  	  there is none.
+    *  -  ``MEDIA_PAD_FL_INTERNAL_SOURCE``
+       -  This flag indicates an internal pad that has no external
+	  connections. Such a pad may not be connected with a link. The internal

"must not"? Or "shall not"?

I think "may not" is appropriate. I'd be fine with shall, too, albeit it wouldn't change the meaning in practice.

Signature · 1 line
Sakari Ailus
#4 lore raw

Re: [RFC 1/7] media: mc: Add INTERNAL_SOURCE pad type flag

Quoted 38 lines · Sakari Ailus

Hi Tomi,

Thank you for the review.

Quoted 24 lines · Tomi Valkeinen
Quoted 23 lines · Sakari Ailus

Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.

Also prevent creating links to pads that have been flagged as internal.

---
   .../userspace-api/media/mediactl/media-types.rst          | 7 +++++++
   drivers/media/mc/mc-entity.c                              | 8 +++++++-
   include/uapi/linux/media.h                                | 1 +
   3 files changed, 15 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/mediactl/media-types.rst +1 -0
index 0ffeece1e0c8..c724139ad46c 100644
@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
   .. _MEDIA-PAD-FL-SINK:
   .. _MEDIA-PAD-FL-SOURCE:
   .. _MEDIA-PAD-FL-MUST-CONNECT:
+.. _MEDIA-PAD-FL-INTERNAL-SOURCE:
   .. flat-table:: Media pad flags
       :header-rows:  0
Quoted 8 lines

Documentation/userspace-api/media/mediactl/media-types.rst Types and flags used to represent the media graph elements · line 387

Quoted 6 lines
Documentation/userspace-api/media/mediactl/media-types.rst +3 -0
@@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements
   	  when this flag isn't set; the absence of the flag doesn't imply
   	  there is none.
+    *  -  ``MEDIA_PAD_FL_INTERNAL_SOURCE``
+       -  This flag indicates an internal pad that has no external
+	  connections. Such a pad may not be connected with a link. The internal

"must not"? Or "shall not"?

Quoted 2 lines

I think "may not" is appropriate. I'd be fine with shall, too, albeit it wouldn't change the meaning in practice.

Ok. I don't speak standardize, and I'm not a native English speaker. But to me "may not be connected" sounds like it is possibly not connected, but also that it is possible for it to be connected.

Tomi

author maintainer
#5 lore raw

Re: [RFC 1/7] media: mc: Add INTERNAL_SOURCE pad type flag

Quoted 39 lines · Tomi Valkeinen
Quoted 38 lines · Sakari Ailus

Hi Tomi,

Thank you for the review.

Quoted 32 lines · Tomi Valkeinen
Quoted 24 lines · Sakari Ailus

Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.

Also prevent creating links to pads that have been flagged as internal.

---
   .../userspace-api/media/mediactl/media-types.rst          | 7 +++++++
   drivers/media/mc/mc-entity.c                              | 8 +++++++-
   include/uapi/linux/media.h                                | 1 +
   3 files changed, 15 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/mediactl/media-types.rst +1 -0
index 0ffeece1e0c8..c724139ad46c 100644
@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
   .. _MEDIA-PAD-FL-SINK:
   .. _MEDIA-PAD-FL-SOURCE:
   .. _MEDIA-PAD-FL-MUST-CONNECT:
+.. _MEDIA-PAD-FL-INTERNAL-SOURCE:
   .. flat-table:: Media pad flags
       :header-rows:  0

Documentation/userspace-api/media/mediactl/media-types.rst Types and flags used to represent the media graph elements · line 387

Quoted 5 lines
Documentation/userspace-api/media/mediactl/media-types.rst +3 -0
@@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements
   	  when this flag isn't set; the absence of the flag doesn't imply
   	  there is none.
+    *  -  ``MEDIA_PAD_FL_INTERNAL_SOURCE``
+       -  This flag indicates an internal pad that has no external
+	  connections. Such a pad may not be connected with a link. The internal

"must not"? Or "shall not"?

Quoted 6 lines
Quoted 2 lines

I think "may not" is appropriate. I'd be fine with shall, too, albeit it wouldn't change the meaning in practice.

Ok. I don't speak standardize, and I'm not a native English speaker. But to me "may not be connected" sounds like it is possibly not connected, but also that it is possible for it to be connected.

Ah, yes, I think you're actually right. But that is what I thought when writing the text. ;-) Shall we use shall, then?

Signature · 1 line
Sakari Ailus
maintainer
#6 lore raw

Re: [RFC 1/7] media: mc: Add INTERNAL_SOURCE pad type flag

In reply to Sakari Ailus

Hi Sakari,

Thank you for the patch.

Quoted 25 lines · Sakari Ailus

Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.

Also prevent creating links to pads that have been flagged as internal.

---
 .../userspace-api/media/mediactl/media-types.rst          | 7 +++++++
 drivers/media/mc/mc-entity.c                              | 8 +++++++-
 include/uapi/linux/media.h                                | 1 +
 3 files changed, 15 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/mediactl/media-types.rst +1 -0
index 0ffeece1e0c8..c724139ad46c 100644
@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
 .. _MEDIA-PAD-FL-SINK:
 .. _MEDIA-PAD-FL-SOURCE:
 .. _MEDIA-PAD-FL-MUST-CONNECT:
+.. _MEDIA-PAD-FL-INTERNAL-SOURCE:
 
 .. flat-table:: Media pad flags
     :header-rows:  0

Documentation/userspace-api/media/mediactl/media-types.rst Types and flags used to represent the media graph elements · line 389

Quoted 7 lines
Documentation/userspace-api/media/mediactl/media-types.rst +4 -0
@@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements
 	  when this flag isn't set; the absence of the flag doesn't imply
 	  there is none.
 
+    *  -  ``MEDIA_PAD_FL_INTERNAL_SOURCE``
+       -  This flag indicates an internal pad that has no external
+	  connections. Such a pad may not be connected with a link. The internal
+	  flag indicates that the stream either starts or ends in the

There's no mention of "stream" (with this meaning) anywhere in the MC API documentation, so you will need to explain streams first.

Apart from that, and addressing Tomi's comment, this patch seems OK.

Quoted 43 lines
Documentation/userspace-api/media/mediactl/media-types.rst +2 -0
+	  entity. For a given pad, the INTERNAL_SOURCE flag may not be set if
+	  either SINK or SOURCE flags is set.
 
 One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
 must be set for every pad.
drivers/media/mc/mc-entity.c +7 -1
index ef34ddd715c9..ed99160a2487 100644
@@ -1062,7 +1062,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
 
 	for (i = 0; i < entity->num_pads; i++) {
 		if ((entity->pads[i].flags &
-		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
+		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
+		      MEDIA_PAD_FL_INTERNAL_SOURCE)) != pad_type)
 			continue;
 
 		if (entity->pads[i].sig_type == sig_type)
@@ -1087,6 +1088,11 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
 		return -EINVAL;
 	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
 		return -EINVAL;
+	if (WARN_ON(sink->pads[sink_pad].flags &
+		    MEDIA_PAD_FL_INTERNAL_SOURCE) ||
+	    WARN_ON(source->pads[source_pad].flags &
+		    MEDIA_PAD_FL_INTERNAL_SOURCE))
+		return -EINVAL;
 
 	link = media_add_link(&source->links);
 	if (link == NULL)
include/uapi/linux/media.h +1 -0
index edb8dfef9eba..0e2577e8b425 100644
@@ -208,6 +208,7 @@ struct media_entity_desc {
 #define MEDIA_PAD_FL_SINK			(1U << 0)
 #define MEDIA_PAD_FL_SOURCE			(1U << 1)
 #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
+#define MEDIA_PAD_FL_INTERNAL_SOURCE		(1U << 3)
 
 struct media_pad_desc {
 	__u32 entity;		/* entity ID */
Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#7 lore raw

Re: [RFC 1/7] media: mc: Add INTERNAL_SOURCE pad type flag

Hi Laurent,

Quoted 33 lines · Laurent Pinchart

Hi Sakari,

Thank you for the patch.

Quoted 28 lines · Sakari Ailus

Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.

Also prevent creating links to pads that have been flagged as internal.

---
 .../userspace-api/media/mediactl/media-types.rst          | 7 +++++++
 drivers/media/mc/mc-entity.c                              | 8 +++++++-
 include/uapi/linux/media.h                                | 1 +
 3 files changed, 15 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/mediactl/media-types.rst +1 -0
index 0ffeece1e0c8..c724139ad46c 100644
@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
 .. _MEDIA-PAD-FL-SINK:
 .. _MEDIA-PAD-FL-SOURCE:
 .. _MEDIA-PAD-FL-MUST-CONNECT:
+.. _MEDIA-PAD-FL-INTERNAL-SOURCE:
 
 .. flat-table:: Media pad flags
     :header-rows:  0
@@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements
 	  when this flag isn't set; the absence of the flag doesn't imply
 	  there is none.
 
Quoted 7 lines

Documentation/userspace-api/media/mediactl/media-types.rst Types and flags used to represent the media graph elements · line 389

Quoted 4 lines
Documentation/userspace-api/media/mediactl/media-types.rst +4 -0
+    *  -  ``MEDIA_PAD_FL_INTERNAL_SOURCE``
+       -  This flag indicates an internal pad that has no external
+	  connections. Such a pad may not be connected with a link. The internal
+	  flag indicates that the stream either starts or ends in the

There's no mention of "stream" (with this meaning) anywhere in the MC API documentation, so you will need to explain streams first.

I'll add it to glossary and add a reference here.

Quoted 47 lines

Apart from that, and addressing Tomi's comment, this patch seems OK.

Quoted 43 lines
Documentation/userspace-api/media/mediactl/media-types.rst +2 -0
+	  entity. For a given pad, the INTERNAL_SOURCE flag may not be set if
+	  either SINK or SOURCE flags is set.
 
 One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
 must be set for every pad.
drivers/media/mc/mc-entity.c +7 -1
index ef34ddd715c9..ed99160a2487 100644
@@ -1062,7 +1062,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
 
 	for (i = 0; i < entity->num_pads; i++) {
 		if ((entity->pads[i].flags &
-		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
+		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
+		      MEDIA_PAD_FL_INTERNAL_SOURCE)) != pad_type)
 			continue;
 
 		if (entity->pads[i].sig_type == sig_type)
@@ -1087,6 +1088,11 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
 		return -EINVAL;
 	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
 		return -EINVAL;
+	if (WARN_ON(sink->pads[sink_pad].flags &
+		    MEDIA_PAD_FL_INTERNAL_SOURCE) ||
+	    WARN_ON(source->pads[source_pad].flags &
+		    MEDIA_PAD_FL_INTERNAL_SOURCE))
+		return -EINVAL;
 
 	link = media_add_link(&source->links);
 	if (link == NULL)
include/uapi/linux/media.h +1 -0
index edb8dfef9eba..0e2577e8b425 100644
@@ -208,6 +208,7 @@ struct media_entity_desc {
 #define MEDIA_PAD_FL_SINK			(1U << 0)
 #define MEDIA_PAD_FL_SOURCE			(1U << 1)
 #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
+#define MEDIA_PAD_FL_INTERNAL_SOURCE		(1U << 3)
 
 struct media_pad_desc {
 	__u32 entity;		/* entity ID */
Signature · 3 lines
Regards,

Sakari Ailus
#8 lore raw

Re: [RFC 1/7] media: mc: Add INTERNAL_SOURCE pad type flag

In reply to Sakari Ailus
Quoted 25 lines · Sakari Ailus

Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.

Also prevent creating links to pads that have been flagged as internal.

---
 .../userspace-api/media/mediactl/media-types.rst          | 7 +++++++
 drivers/media/mc/mc-entity.c                              | 8 +++++++-
 include/uapi/linux/media.h                                | 1 +
 3 files changed, 15 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/mediactl/media-types.rst +1 -0
index 0ffeece1e0c8..c724139ad46c 100644
@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
 .. _MEDIA-PAD-FL-SINK:
 .. _MEDIA-PAD-FL-SOURCE:
 .. _MEDIA-PAD-FL-MUST-CONNECT:
+.. _MEDIA-PAD-FL-INTERNAL-SOURCE:
 
 .. flat-table:: Media pad flags
     :header-rows:  0

Documentation/userspace-api/media/mediactl/media-types.rst Types and flags used to represent the media graph elements · line 387

Quoted 5 lines
Documentation/userspace-api/media/mediactl/media-types.rst +2 -0
@@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements
 	  when this flag isn't set; the absence of the flag doesn't imply
 	  there is none.
 
+    *  -  ``MEDIA_PAD_FL_INTERNAL_SOURCE``
+       -  This flag indicates an internal pad that has no external

'internal pad' or 'internal source pad'?

Documentation/userspace-api/media/mediactl/media-types.rst Types and flags used to represent the media graph elements

Quoted 3 lines
Documentation/userspace-api/media/mediactl/media-types.rst +3 -0
+	  connections. Such a pad may not be connected with a link. The internal
+	  flag indicates that the stream either starts or ends in the
+	  entity.

This sentence makes no sense. The flag is for a pad, but then this sentence talks about streams and entities and doesn't relate that to the pad. Also, the flag is INTERNAL_SOURCE, and reading 'stream ends' suggests that it should be INTERNAL_SINK in that case.

I think this needs to be more elaborate and ideally add some examples of the use-case(s) (or refer to an example elsewhere).

For a given pad, the INTERNAL_SOURCE flag may not be set if

Documentation/userspace-api/media/mediactl/media-types.rst Types and flags used to represent the media graph elements · line 393

Quoted 3 lines
Documentation/userspace-api/media/mediactl/media-types.rst +1 -0
+	  either SINK or SOURCE flags is set.
 
 One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``

and 'MEDIA_PAD_FL_INTERNAL_SOURCE' as well?

Quoted 32 lines
Documentation/userspace-api/media/mediactl/media-types.rst +0 -0
 must be set for every pad.
drivers/media/mc/mc-entity.c +7 -1
index ef34ddd715c9..ed99160a2487 100644
@@ -1062,7 +1062,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
 
 	for (i = 0; i < entity->num_pads; i++) {
 		if ((entity->pads[i].flags &
-		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
+		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
+		      MEDIA_PAD_FL_INTERNAL_SOURCE)) != pad_type)
 			continue;
 
 		if (entity->pads[i].sig_type == sig_type)
@@ -1087,6 +1088,11 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
 		return -EINVAL;
 	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
 		return -EINVAL;
+	if (WARN_ON(sink->pads[sink_pad].flags &
+		    MEDIA_PAD_FL_INTERNAL_SOURCE) ||
+	    WARN_ON(source->pads[source_pad].flags &
+		    MEDIA_PAD_FL_INTERNAL_SOURCE))
+		return -EINVAL;
 
 	link = media_add_link(&source->links);
 	if (link == NULL)
include/uapi/linux/media.h +0 -0
index edb8dfef9eba..0e2577e8b425 100644

include/uapi/linux/media.h struct media_entity_desc { · line 214

Quoted 7 lines
include/uapi/linux/media.h +1 -0
@@ -208,6 +208,7 @@ struct media_entity_desc {
 #define MEDIA_PAD_FL_SINK			(1U << 0)
 #define MEDIA_PAD_FL_SOURCE			(1U << 1)
 #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
+#define MEDIA_PAD_FL_INTERNAL_SOURCE		(1U << 3)
 
 struct media_pad_desc {
 	__u32 entity;		/* entity ID */

Regards,

Hans

author maintainer
#9 lore raw

Re: [RFC 1/7] media: mc: Add INTERNAL_SOURCE pad type flag

Hi Hans,

Thanks for the review.

Quoted 34 lines · Hans Verkuil
Quoted 25 lines · Sakari Ailus

Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.

Also prevent creating links to pads that have been flagged as internal.

---
 .../userspace-api/media/mediactl/media-types.rst          | 7 +++++++
 drivers/media/mc/mc-entity.c                              | 8 +++++++-
 include/uapi/linux/media.h                                | 1 +
 3 files changed, 15 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/mediactl/media-types.rst +1 -0
index 0ffeece1e0c8..c724139ad46c 100644
@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
 .. _MEDIA-PAD-FL-SINK:
 .. _MEDIA-PAD-FL-SOURCE:
 .. _MEDIA-PAD-FL-MUST-CONNECT:
+.. _MEDIA-PAD-FL-INTERNAL-SOURCE:
 
 .. flat-table:: Media pad flags
     :header-rows:  0

Documentation/userspace-api/media/mediactl/media-types.rst Types and flags used to represent the media graph elements · line 387

Quoted 5 lines
Documentation/userspace-api/media/mediactl/media-types.rst +2 -0
@@ -382,6 +383,12 @@ Types and flags used to represent the media graph elements
 	  when this flag isn't set; the absence of the flag doesn't imply
 	  there is none.
 
+    *  -  ``MEDIA_PAD_FL_INTERNAL_SOURCE``
+       -  This flag indicates an internal pad that has no external

'internal pad' or 'internal source pad'?

Quoted 6 lines

Documentation/userspace-api/media/mediactl/media-types.rst Types and flags used to represent the media graph elements

Quoted 3 lines
Documentation/userspace-api/media/mediactl/media-types.rst +3 -0
+	  connections. Such a pad may not be connected with a link. The internal
+	  flag indicates that the stream either starts or ends in the
+	  entity.

This sentence makes no sense. The flag is for a pad, but then this sentence talks about streams and entities and doesn't relate that to the pad. Also,

Streams are between pads, but are visible on V4L2 only while this is MC related. Documentation-wise this is a bit awkward.

Quoted 2 lines

the flag is INTERNAL_SOURCE, and reading 'stream ends' suggests that it should be INTERNAL_SINK in that case.

When it comes to routing, it uses fields related to the sink pad, but this is a source.

Based on review of the other patches, it seems both Tomi and Laurent would prefer a plain INTERNAL flag which then would be used in conjunction with the existing SINK flag, to note an internal source. Routing-wise this is a cleaner option as this is associated with the sink related fields in any case.

Quoted 54 lines

I think this needs to be more elaborate and ideally add some examples of the use-case(s) (or refer to an example elsewhere).

For a given pad, the INTERNAL_SOURCE flag may not be set if

Documentation/userspace-api/media/mediactl/media-types.rst Types and flags used to represent the media graph elements · line 393

Quoted 3 lines
Documentation/userspace-api/media/mediactl/media-types.rst +1 -0
+	  either SINK or SOURCE flags is set.
 
 One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``

and 'MEDIA_PAD_FL_INTERNAL_SOURCE' as well?

Quoted 32 lines
Documentation/userspace-api/media/mediactl/media-types.rst +0 -0
 must be set for every pad.
drivers/media/mc/mc-entity.c +7 -1
index ef34ddd715c9..ed99160a2487 100644
@@ -1062,7 +1062,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
 
 	for (i = 0; i < entity->num_pads; i++) {
 		if ((entity->pads[i].flags &
-		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
+		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
+		      MEDIA_PAD_FL_INTERNAL_SOURCE)) != pad_type)
 			continue;
 
 		if (entity->pads[i].sig_type == sig_type)
@@ -1087,6 +1088,11 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
 		return -EINVAL;
 	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
 		return -EINVAL;
+	if (WARN_ON(sink->pads[sink_pad].flags &
+		    MEDIA_PAD_FL_INTERNAL_SOURCE) ||
+	    WARN_ON(source->pads[source_pad].flags &
+		    MEDIA_PAD_FL_INTERNAL_SOURCE))
+		return -EINVAL;
 
 	link = media_add_link(&source->links);
 	if (link == NULL)
include/uapi/linux/media.h +0 -0
index edb8dfef9eba..0e2577e8b425 100644

include/uapi/linux/media.h struct media_entity_desc { · line 214

Quoted 7 lines
include/uapi/linux/media.h +1 -0
@@ -208,6 +208,7 @@ struct media_entity_desc {
 #define MEDIA_PAD_FL_SINK			(1U << 0)
 #define MEDIA_PAD_FL_SOURCE			(1U << 1)
 #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
+#define MEDIA_PAD_FL_INTERNAL_SOURCE		(1U << 3)
 
 struct media_pad_desc {
 	__u32 entity;		/* entity ID */

Regards,

Hans

Signature · 3 lines
Kind regards,

Sakari Ailus
author maintainer
#10 lore raw

RFC 2/7 media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs

In reply to Sakari Ailus

Take the new INTERNAL_SOURCE pad flag into account in validating routing IOCTL argument. Effectively this is a SINK pad in this respect. Due to the union there's no need to check for the particular name.

---
 drivers/media/v4l2-core/v4l2-subdev.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/media/v4l2-core/v4l2-subdev.c +4 -2
index 2ec179cd1264..6312fc2bf1f0 100644
@@ -906,7 +906,8 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 				return -EINVAL;
 
 			if (!(pads[route->sink_pad].flags &
-			      MEDIA_PAD_FL_SINK))
+			      (MEDIA_PAD_FL_SINK |
+			       MEDIA_PAD_FL_INTERNAL_SOURCE)))
 				return -EINVAL;
 
 			if (route->source_pad >= sd->entity.num_pads)
@@ -1787,7 +1788,8 @@ int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
 
 		/* Validate the sink and source pad numbers. */
 		if (route->sink_pad >= sd->entity.num_pads ||
-		    !(sd->entity.pads[route->sink_pad].flags & MEDIA_PAD_FL_SINK)) {
+		    !(sd->entity.pads[route->sink_pad].flags &
+		      (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL_SOURCE))) {
 			dev_dbg(sd->dev, "route %u sink (%u) is not a sink pad\n",
 				i, route->sink_pad);
 			goto out;
Signature · 1 line
2.30.2
#11 lore raw

Re: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs

Quoted 3 lines · Sakari Ailus

Take the new INTERNAL_SOURCE pad flag into account in validating routing IOCTL argument. Effectively this is a SINK pad in this respect. Due to the union there's no need to check for the particular name.

What union? The one you add in the next patch?

As a concept the internal source pads sound good, and they work in practice in my tests. But this union is what grates me a bit. We have a flag, MEDIA_PAD_FL_INTERNAL_SOURCE, which tells which field of the union to use, and then we go and do not use the new union field. Well, and also the naming, as we normally have source and sink pads, but now we also have internal source pads, which are actually identical to sink pads...

I understand the idea and reasoning, but the two points above do confuse me and I'm sure would confuse others also.

I wonder if it would be less or more confusing to simplify this by just adding a MEDIA_PAD_FL_INTERNAL, which could be applied to either a source or a sink pad, and would prevent linking to it. The flag would indicate that the stream from/to that pad is generated/consumed internally. (I don't know when you would need an internal pad to consume data, but... who knows, the need might pop up =).

That would mean that an "internal sink pad" would generate a data stream, i.e. it's a "source", but I think a normal sink pad is almost the same anyway: when considering entity's internal routing, the normal sink pad is a "source", and the same logic would apply with the internal pads too.

An internal sink pad that generates a stream is a bit more confusing than an internal source pad, but I think that confusion is less than the rest of the confusion in the code-side that comes with the internal source pads.

Tomi

Quoted 29 lines
---
  drivers/media/v4l2-core/v4l2-subdev.c | 6 ++++--
  1 file changed, 4 insertions(+), 2 deletions(-)
drivers/media/v4l2-core/v4l2-subdev.c +4 -2
index 2ec179cd1264..6312fc2bf1f0 100644
@@ -906,7 +906,8 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
  				return -EINVAL;
 
  			if (!(pads[route->sink_pad].flags &
-			      MEDIA_PAD_FL_SINK))
+			      (MEDIA_PAD_FL_SINK |
+			       MEDIA_PAD_FL_INTERNAL_SOURCE)))
  				return -EINVAL;
 
  			if (route->source_pad >= sd->entity.num_pads)
@@ -1787,7 +1788,8 @@ int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
 
  		/* Validate the sink and source pad numbers. */
  		if (route->sink_pad >= sd->entity.num_pads ||
-		    !(sd->entity.pads[route->sink_pad].flags & MEDIA_PAD_FL_SINK)) {
+		    !(sd->entity.pads[route->sink_pad].flags &
+		      (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL_SOURCE))) {
  			dev_dbg(sd->dev, "route %u sink (%u) is not a sink pad\n",
  				i, route->sink_pad);
  			goto out;
author maintainer
#12 lore raw

Re: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs

Moi,

Quoted 6 lines · Tomi Valkeinen
Quoted 3 lines · Sakari Ailus

Take the new INTERNAL_SOURCE pad flag into account in validating routing IOCTL argument. Effectively this is a SINK pad in this respect. Due to the union there's no need to check for the particular name.

What union? The one you add in the next patch?

Oops. I think we can reorder the patches.

Quoted 7 lines

As a concept the internal source pads sound good, and they work in practice in my tests. But this union is what grates me a bit. We have a flag, MEDIA_PAD_FL_INTERNAL_SOURCE, which tells which field of the union to use, and then we go and do not use the new union field. Well, and also the naming, as we normally have source and sink pads, but now we also have internal source pads, which are actually identical to sink pads...

The union still should be used by the user space. We're testing flags here and those flags are the same independently of the INTERNAL_SOURCE flag.

I'm fine by not adding that union though, but for the user space I think it's better we have it: explaining that the sink_pad has a different meaning if that flag is set is harder than making it a union member.

Quoted 10 lines

I understand the idea and reasoning, but the two points above do confuse me and I'm sure would confuse others also.

I wonder if it would be less or more confusing to simplify this by just adding a MEDIA_PAD_FL_INTERNAL, which could be applied to either a source or a sink pad, and would prevent linking to it. The flag would indicate that the stream from/to that pad is generated/consumed internally. (I don't know when you would need an internal pad to consume data, but... who knows, the need might pop up =).

This is another option. But I envision there will be more compatibility issues. Although... generally using such hardware requires knowledge of the device, and we haven't obviously had any support for devices needing this functionality in the tree. So in the end it might not matter much.

Quoted 9 lines

That would mean that an "internal sink pad" would generate a data stream, i.e. it's a "source", but I think a normal sink pad is almost the same
anyway: when considering entity's internal routing, the normal sink pad is a "source", and the same logic would apply with the internal pads too.

An internal sink pad that generates a stream is a bit more confusing than an internal source pad, but I think that confusion is less than the rest of the confusion in the code-side that comes with the internal source pads.

I prefer having the possible sources of the confusion in the framework than in the drivers. Therefore I think INTERNAL_SOURCE flag is a (slightly) better option.

I wonder what Llaurent thinks.

Signature · 3 lines
Kind regards,

Sakari Ailus
maintainer
#13 lore raw

Re: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs

Quoted 1 line · Sakari Ailus

Moi,

こんにちは

Quoted 48 lines · Tomi Valkeinen
Quoted 6 lines · Sakari Ailus
Quoted 3 lines

Take the new INTERNAL_SOURCE pad flag into account in validating routing IOCTL argument. Effectively this is a SINK pad in this respect. Due to the union there's no need to check for the particular name.

What union? The one you add in the next patch?

Oops. I think we can reorder the patches.

Quoted 6 lines

As a concept the internal source pads sound good, and they work in practice in my tests. But this union is what grates me a bit. We have a flag, MEDIA_PAD_FL_INTERNAL_SOURCE, which tells which field of the union to use, and then we go and do not use the new union field. Well, and also the naming, as we normally have source and sink pads, but now we also have internal source pads, which are actually identical to sink pads...

The union still should be used by the user space. We're testing flags here and those flags are the same independently of the INTERNAL_SOURCE flag.

I'm fine by not adding that union though, but for the user space I think it's better we have it: explaining that the sink_pad has a different meaning if that flag is set is harder than making it a union member.

Quoted 9 lines

I understand the idea and reasoning, but the two points above do confuse me and I'm sure would confuse others also.

I wonder if it would be less or more confusing to simplify this by just adding a MEDIA_PAD_FL_INTERNAL, which could be applied to either a source or a sink pad, and would prevent linking to it. The flag would indicate that the stream from/to that pad is generated/consumed internally. (I don't know when you would need an internal pad to consume data, but... who knows, the need might pop up =).

This is another option. But I envision there will be more compatibility issues. Although... generally using such hardware requires knowledge of the device, and we haven't obviously had any support for devices needing this functionality in the tree. So in the end it might not matter much.

Quoted 8 lines

That would mean that an "internal sink pad" would generate a data stream, i.e. it's a "source", but I think a normal sink pad is almost the same
anyway: when considering entity's internal routing, the normal sink pad is a "source", and the same logic would apply with the internal pads too.

An internal sink pad that generates a stream is a bit more confusing than an internal source pad, but I think that confusion is less than the rest of the confusion in the code-side that comes with the internal source pads.

Quoted 5 lines

I prefer having the possible sources of the confusion in the framework than in the drivers. Therefore I think INTERNAL_SOURCE flag is a (slightly) better option.

I wonder what Llaurent thinks.

I like the idea of a MEDIA_PAD_FL_INTERNAL flag. That's actually how I read patch 1/7, I didn't notice it used MEDIA_PAD_FL_INTERNAL*_SOURCE* :-)

We could define MEDIA_PAD_FL_INTERNAL_SOURCE

#define MEDIA_PAD_FL_INTERNAL_SOURCE (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL)

but I'm not sure it would be less confusing.

Regarding isolating the sources of confusion in the framework rather than spreading them through drivers, I can't disagree, but I think that, for raw camera sensors at least, the best option would be to create a new camera sensor object with a much more tailored API than v4l2_subdev (and of course wrapping that new object in a v4l2_subdev in the framework). This won't address the other types of drivers, but I'm not sure we'll get any in the foreseable future.

Signature · 3 lines
Regards,

Laurent Pinchart
maintainer
#14 lore raw

Re: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs

Quoted 59 lines · Laurent Pinchart
Quoted 1 line · Sakari Ailus

Moi,

こんにちは

Quoted 48 lines · Tomi Valkeinen
Quoted 6 lines · Sakari Ailus
Quoted 3 lines

Take the new INTERNAL_SOURCE pad flag into account in validating routing IOCTL argument. Effectively this is a SINK pad in this respect. Due to the union there's no need to check for the particular name.

What union? The one you add in the next patch?

Oops. I think we can reorder the patches.

Quoted 6 lines

As a concept the internal source pads sound good, and they work in practice in my tests. But this union is what grates me a bit. We have a flag, MEDIA_PAD_FL_INTERNAL_SOURCE, which tells which field of the union to use, and then we go and do not use the new union field. Well, and also the naming, as we normally have source and sink pads, but now we also have internal source pads, which are actually identical to sink pads...

The union still should be used by the user space. We're testing flags here and those flags are the same independently of the INTERNAL_SOURCE flag.

I'm fine by not adding that union though, but for the user space I think it's better we have it: explaining that the sink_pad has a different meaning if that flag is set is harder than making it a union member.

Quoted 9 lines

I understand the idea and reasoning, but the two points above do confuse me and I'm sure would confuse others also.

I wonder if it would be less or more confusing to simplify this by just adding a MEDIA_PAD_FL_INTERNAL, which could be applied to either a source or a sink pad, and would prevent linking to it. The flag would indicate that the stream from/to that pad is generated/consumed internally. (I don't know when you would need an internal pad to consume data, but... who knows, the need might pop up =).

This is another option. But I envision there will be more compatibility issues. Although... generally using such hardware requires knowledge of the device, and we haven't obviously had any support for devices needing this functionality in the tree. So in the end it might not matter much.

Quoted 8 lines

That would mean that an "internal sink pad" would generate a data stream, i.e. it's a "source", but I think a normal sink pad is almost the same
anyway: when considering entity's internal routing, the normal sink pad is a "source", and the same logic would apply with the internal pads too.

An internal sink pad that generates a stream is a bit more confusing than an internal source pad, but I think that confusion is less than the rest of the confusion in the code-side that comes with the internal source pads.

Quoted 5 lines

I prefer having the possible sources of the confusion in the framework than in the drivers. Therefore I think INTERNAL_SOURCE flag is a (slightly) better option.

I wonder what Llaurent thinks.

Quoted 7 lines

I like the idea of a MEDIA_PAD_FL_INTERNAL flag. That's actually how I read patch 1/7, I didn't notice it used MEDIA_PAD_FL_INTERNAL*_SOURCE* :-)

We could define MEDIA_PAD_FL_INTERNAL_SOURCE

#define MEDIA_PAD_FL_INTERNAL_SOURCE (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL)

One option would be to find terms different from sink and source in this case. It would generate less confusion to map (e.g., not a really good name) MEDIA_PAD_FL_INTERNAL_PRODUCER to MEDIA_PAD_FL_SINK and to the sink_pad field than using MEDIA_PAD_FL_INTERNAL_SOURCE.

Quoted 9 lines

but I'm not sure it would be less confusing.

Regarding isolating the sources of confusion in the framework rather than spreading them through drivers, I can't disagree, but I think that, for raw camera sensors at least, the best option would be to create a new camera sensor object with a much more tailored API than v4l2_subdev (and of course wrapping that new object in a v4l2_subdev in the framework). This won't address the other types of drivers, but I'm not sure we'll get any in the foreseable future.

Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#15 lore raw

Re: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs

Heippa Tomi, Laurent,

Quoted 64 lines · Laurent Pinchart
Quoted 63 lines · Laurent Pinchart
Quoted 1 line · Sakari Ailus

Moi,

こんにちは

Quoted 48 lines · Tomi Valkeinen
Quoted 6 lines · Sakari Ailus
Quoted 3 lines

Take the new INTERNAL_SOURCE pad flag into account in validating routing IOCTL argument. Effectively this is a SINK pad in this respect. Due to the union there's no need to check for the particular name.

What union? The one you add in the next patch?

Oops. I think we can reorder the patches.

Quoted 6 lines

As a concept the internal source pads sound good, and they work in practice in my tests. But this union is what grates me a bit. We have a flag, MEDIA_PAD_FL_INTERNAL_SOURCE, which tells which field of the union to use, and then we go and do not use the new union field. Well, and also the naming, as we normally have source and sink pads, but now we also have internal source pads, which are actually identical to sink pads...

The union still should be used by the user space. We're testing flags here and those flags are the same independently of the INTERNAL_SOURCE flag.

I'm fine by not adding that union though, but for the user space I think it's better we have it: explaining that the sink_pad has a different meaning if that flag is set is harder than making it a union member.

Quoted 9 lines

I understand the idea and reasoning, but the two points above do confuse me and I'm sure would confuse others also.

I wonder if it would be less or more confusing to simplify this by just adding a MEDIA_PAD_FL_INTERNAL, which could be applied to either a source or a sink pad, and would prevent linking to it. The flag would indicate that the stream from/to that pad is generated/consumed internally. (I don't know when you would need an internal pad to consume data, but... who knows, the need might pop up =).

This is another option. But I envision there will be more compatibility issues. Although... generally using such hardware requires knowledge of the device, and we haven't obviously had any support for devices needing this functionality in the tree. So in the end it might not matter much.

Quoted 8 lines

That would mean that an "internal sink pad" would generate a data stream, i.e. it's a "source", but I think a normal sink pad is almost the same
anyway: when considering entity's internal routing, the normal sink pad is a "source", and the same logic would apply with the internal pads too.

An internal sink pad that generates a stream is a bit more confusing than an internal source pad, but I think that confusion is less than the rest of the confusion in the code-side that comes with the internal source pads.

Quoted 5 lines

I prefer having the possible sources of the confusion in the framework than in the drivers. Therefore I think INTERNAL_SOURCE flag is a (slightly) better option.

I wonder what Llaurent thinks.

I like the idea of a MEDIA_PAD_FL_INTERNAL flag. That's actually how I read patch 1/7, I didn't notice it used MEDIA_PAD_FL_INTERNAL*_SOURCE* :-)

Quoted 8 lines
Quoted 3 lines

We could define MEDIA_PAD_FL_INTERNAL_SOURCE

#define MEDIA_PAD_FL_INTERNAL_SOURCE (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL)

One option would be to find terms different from sink and source in this case. It would generate less confusion to map (e.g., not a really good name) MEDIA_PAD_FL_INTERNAL_PRODUCER to MEDIA_PAD_FL_SINK and to the sink_pad field than using MEDIA_PAD_FL_INTERNAL_SOURCE.

I don't think this helps as you'd still be accessing the sink pad related fields that are there for another purpose.

Alternatively I'd have the (plain) INTERNAL flag and drop the union, without masking or adding compound flags.

I can switch to that if you promise not to change your mind again. ;-)

Quoted 11 lines
Quoted 10 lines

but I'm not sure it would be less confusing.

Regarding isolating the sources of confusion in the framework rather than spreading them through drivers, I can't disagree, but I think that, for raw camera sensors at least, the best option would be to create a new camera sensor object with a much more tailored API than v4l2_subdev (and of course wrapping that new object in a v4l2_subdev in the framework). This won't address the other types of drivers, but I'm not sure we'll get any in the foreseable future.

Signature · 3 lines
Terveisin,

Sakari Ailus
maintainer
#16 lore raw

Re: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs

Hi Sakari,

Quoted 70 lines · Sakari Ailus
Quoted 60 lines · Laurent Pinchart
Quoted 59 lines · Laurent Pinchart
Quoted 48 lines · Sakari Ailus
Quoted 6 lines · Tomi Valkeinen
Quoted 3 lines · Sakari Ailus

Take the new INTERNAL_SOURCE pad flag into account in validating routing IOCTL argument. Effectively this is a SINK pad in this respect. Due to the union there's no need to check for the particular name.

What union? The one you add in the next patch?

Oops. I think we can reorder the patches.

Quoted 6 lines

As a concept the internal source pads sound good, and they work in practice in my tests. But this union is what grates me a bit. We have a flag, MEDIA_PAD_FL_INTERNAL_SOURCE, which tells which field of the union to use, and then we go and do not use the new union field. Well, and also the naming, as we normally have source and sink pads, but now we also have internal source pads, which are actually identical to sink pads...

The union still should be used by the user space. We're testing flags here and those flags are the same independently of the INTERNAL_SOURCE flag.

I'm fine by not adding that union though, but for the user space I think it's better we have it: explaining that the sink_pad has a different meaning if that flag is set is harder than making it a union member.

Quoted 9 lines

I understand the idea and reasoning, but the two points above do confuse me and I'm sure would confuse others also.

I wonder if it would be less or more confusing to simplify this by just adding a MEDIA_PAD_FL_INTERNAL, which could be applied to either a source or a sink pad, and would prevent linking to it. The flag would indicate that the stream from/to that pad is generated/consumed internally. (I don't know when you would need an internal pad to consume data, but... who knows, the need might pop up =).

This is another option. But I envision there will be more compatibility issues. Although... generally using such hardware requires knowledge of the device, and we haven't obviously had any support for devices needing this functionality in the tree. So in the end it might not matter much.

Quoted 8 lines

That would mean that an "internal sink pad" would generate a data stream, i.e. it's a "source", but I think a normal sink pad is almost the same
anyway: when considering entity's internal routing, the normal sink pad is a "source", and the same logic would apply with the internal pads too.

An internal sink pad that generates a stream is a bit more confusing than an internal source pad, but I think that confusion is less than the rest of the confusion in the code-side that comes with the internal source pads.

Quoted 5 lines

I prefer having the possible sources of the confusion in the framework than in the drivers. Therefore I think INTERNAL_SOURCE flag is a (slightly) better option.

I wonder what Llaurent thinks.

I like the idea of a MEDIA_PAD_FL_INTERNAL flag. That's actually how I read patch 1/7, I didn't notice it used MEDIA_PAD_FL_INTERNAL*_SOURCE* :-)

Quoted 8 lines
Quoted 3 lines

We could define MEDIA_PAD_FL_INTERNAL_SOURCE

#define MEDIA_PAD_FL_INTERNAL_SOURCE (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL)

One option would be to find terms different from sink and source in this case. It would generate less confusion to map (e.g., not a really good name) MEDIA_PAD_FL_INTERNAL_PRODUCER to MEDIA_PAD_FL_SINK and to the sink_pad field than using MEDIA_PAD_FL_INTERNAL_SOURCE.

Quoted 7 lines

I don't think this helps as you'd still be accessing the sink pad related fields that are there for another purpose.

Alternatively I'd have the (plain) INTERNAL flag and drop the union, without masking or adding compound flags.

I can switch to that if you promise not to change your mind again. ;-)

I'll do my best :-) Could you show the impact (if any) it would have on drivers when accessing the routing fields ?

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

but I'm not sure it would be less confusing.

Regarding isolating the sources of confusion in the framework rather than spreading them through drivers, I can't disagree, but I think that, for raw camera sensors at least, the best option would be to create a new camera sensor object with a much more tailored API than v4l2_subdev (and of course wrapping that new object in a v4l2_subdev in the framework). This won't address the other types of drivers, but I'm not sure we'll get any in the foreseable future.

Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#17 lore raw

Re: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs

Hi Laurent,

Quoted 76 lines · Laurent Pinchart

Hi Sakari,

Quoted 73 lines · Sakari Ailus
Quoted 60 lines · Laurent Pinchart
Quoted 59 lines · Laurent Pinchart
Quoted 48 lines · Sakari Ailus
Quoted 6 lines · Tomi Valkeinen
Quoted 3 lines · Sakari Ailus

Take the new INTERNAL_SOURCE pad flag into account in validating routing IOCTL argument. Effectively this is a SINK pad in this respect. Due to the union there's no need to check for the particular name.

What union? The one you add in the next patch?

Oops. I think we can reorder the patches.

Quoted 6 lines

As a concept the internal source pads sound good, and they work in practice in my tests. But this union is what grates me a bit. We have a flag, MEDIA_PAD_FL_INTERNAL_SOURCE, which tells which field of the union to use, and then we go and do not use the new union field. Well, and also the naming, as we normally have source and sink pads, but now we also have internal source pads, which are actually identical to sink pads...

The union still should be used by the user space. We're testing flags here and those flags are the same independently of the INTERNAL_SOURCE flag.

I'm fine by not adding that union though, but for the user space I think it's better we have it: explaining that the sink_pad has a different meaning if that flag is set is harder than making it a union member.

Quoted 9 lines

I understand the idea and reasoning, but the two points above do confuse me and I'm sure would confuse others also.

I wonder if it would be less or more confusing to simplify this by just adding a MEDIA_PAD_FL_INTERNAL, which could be applied to either a source or a sink pad, and would prevent linking to it. The flag would indicate that the stream from/to that pad is generated/consumed internally. (I don't know when you would need an internal pad to consume data, but... who knows, the need might pop up =).

This is another option. But I envision there will be more compatibility issues. Although... generally using such hardware requires knowledge of the device, and we haven't obviously had any support for devices needing this functionality in the tree. So in the end it might not matter much.

Quoted 8 lines

That would mean that an "internal sink pad" would generate a data stream, i.e. it's a "source", but I think a normal sink pad is almost the same
anyway: when considering entity's internal routing, the normal sink pad is a "source", and the same logic would apply with the internal pads too.

An internal sink pad that generates a stream is a bit more confusing than an internal source pad, but I think that confusion is less than the rest of the confusion in the code-side that comes with the internal source pads.

Quoted 5 lines

I prefer having the possible sources of the confusion in the framework than in the drivers. Therefore I think INTERNAL_SOURCE flag is a (slightly) better option.

I wonder what Llaurent thinks.

I like the idea of a MEDIA_PAD_FL_INTERNAL flag. That's actually how I read patch 1/7, I didn't notice it used MEDIA_PAD_FL_INTERNAL*_SOURCE* :-)

Quoted 8 lines
Quoted 3 lines

We could define MEDIA_PAD_FL_INTERNAL_SOURCE

#define MEDIA_PAD_FL_INTERNAL_SOURCE (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL)

One option would be to find terms different from sink and source in this case. It would generate less confusion to map (e.g., not a really good name) MEDIA_PAD_FL_INTERNAL_PRODUCER to MEDIA_PAD_FL_SINK and to the sink_pad field than using MEDIA_PAD_FL_INTERNAL_SOURCE.

I don't think this helps as you'd still be accessing the sink pad related fields that are there for another purpose.

Quoted 7 lines
Quoted 4 lines

Alternatively I'd have the (plain) INTERNAL flag and drop the union, without masking or adding compound flags.

I can switch to that if you promise not to change your mind again. ;-)

I'll do my best :-) Could you show the impact (if any) it would have on drivers when accessing the routing fields ?

I don't think there's much of an impact for the drivers. It's mainly affecting setting up pads for the entities. Tomi?

Signature · 1 line
Sakari Ailus
maintainer
#18 lore raw

Re: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs

Quoted 78 lines · Sakari Ailus
Quoted 77 lines · Laurent Pinchart
Quoted 76 lines · Sakari Ailus
Quoted 60 lines · Laurent Pinchart
Quoted 59 lines · Laurent Pinchart
Quoted 48 lines · Sakari Ailus
Quoted 6 lines · Tomi Valkeinen
Quoted 3 lines · Sakari Ailus

Take the new INTERNAL_SOURCE pad flag into account in validating routing IOCTL argument. Effectively this is a SINK pad in this respect. Due to the union there's no need to check for the particular name.

What union? The one you add in the next patch?

Oops. I think we can reorder the patches.

Quoted 6 lines

As a concept the internal source pads sound good, and they work in practice in my tests. But this union is what grates me a bit. We have a flag, MEDIA_PAD_FL_INTERNAL_SOURCE, which tells which field of the union to use, and then we go and do not use the new union field. Well, and also the naming, as we normally have source and sink pads, but now we also have internal source pads, which are actually identical to sink pads...

The union still should be used by the user space. We're testing flags here and those flags are the same independently of the INTERNAL_SOURCE flag.

I'm fine by not adding that union though, but for the user space I think it's better we have it: explaining that the sink_pad has a different meaning if that flag is set is harder than making it a union member.

Quoted 9 lines

I understand the idea and reasoning, but the two points above do confuse me and I'm sure would confuse others also.

I wonder if it would be less or more confusing to simplify this by just adding a MEDIA_PAD_FL_INTERNAL, which could be applied to either a source or a sink pad, and would prevent linking to it. The flag would indicate that the stream from/to that pad is generated/consumed internally. (I don't know when you would need an internal pad to consume data, but... who knows, the need might pop up =).

This is another option. But I envision there will be more compatibility issues. Although... generally using such hardware requires knowledge of the device, and we haven't obviously had any support for devices needing this functionality in the tree. So in the end it might not matter much.

Quoted 8 lines

That would mean that an "internal sink pad" would generate a data stream, i.e. it's a "source", but I think a normal sink pad is almost the same
anyway: when considering entity's internal routing, the normal sink pad is a "source", and the same logic would apply with the internal pads too.

An internal sink pad that generates a stream is a bit more confusing than an internal source pad, but I think that confusion is less than the rest of the confusion in the code-side that comes with the internal source pads.

Quoted 5 lines

I prefer having the possible sources of the confusion in the framework than in the drivers. Therefore I think INTERNAL_SOURCE flag is a (slightly) better option.

I wonder what Llaurent thinks.

I like the idea of a MEDIA_PAD_FL_INTERNAL flag. That's actually how I read patch 1/7, I didn't notice it used MEDIA_PAD_FL_INTERNAL*_SOURCE* :-)

Quoted 8 lines
Quoted 3 lines

We could define MEDIA_PAD_FL_INTERNAL_SOURCE

#define MEDIA_PAD_FL_INTERNAL_SOURCE (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL)

One option would be to find terms different from sink and source in this case. It would generate less confusion to map (e.g., not a really good name) MEDIA_PAD_FL_INTERNAL_PRODUCER to MEDIA_PAD_FL_SINK and to the sink_pad field than using MEDIA_PAD_FL_INTERNAL_SOURCE.

I don't think this helps as you'd still be accessing the sink pad related fields that are there for another purpose.

Alternatively I'd have the (plain) INTERNAL flag and drop the union, without masking or adding compound flags.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

I can switch to that if you promise not to change your mind again. ;-)

I'll do my best :-) Could you show the impact (if any) it would have on drivers when accessing the routing fields ?

I don't think there's much of an impact for the drivers. It's mainly affecting setting up pads for the entities. Tomi?

I think it would be useful to discuss it with an actual sensor driver using the API.

Signature · 3 lines
Regards,

Laurent Pinchart
#19 lore raw

Re: [RFC 2/7] media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs

In reply to Sakari Ailus
Quoted 24 lines · Sakari Ailus

Take the new INTERNAL_SOURCE pad flag into account in validating routing IOCTL argument. Effectively this is a SINK pad in this respect. Due to the union there's no need to check for the particular name.

---
 drivers/media/v4l2-core/v4l2-subdev.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/media/v4l2-core/v4l2-subdev.c +2 -1
index 2ec179cd1264..6312fc2bf1f0 100644
@@ -906,7 +906,8 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 				return -EINVAL;
 
 			if (!(pads[route->sink_pad].flags &
-			      MEDIA_PAD_FL_SINK))
+			      (MEDIA_PAD_FL_SINK |
+			       MEDIA_PAD_FL_INTERNAL_SOURCE)))
 				return -EINVAL;
 
 			if (route->source_pad >= sd->entity.num_pads)

drivers/media/v4l2-core/v4l2-subdev.c int v4l2_subdev_routing_validate(struct v4l2_subdev *sd, · line 1792

Quoted 6 lines
drivers/media/v4l2-core/v4l2-subdev.c +2 -1
@@ -1787,7 +1788,8 @@ int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
 
 		/* Validate the sink and source pad numbers. */
 		if (route->sink_pad >= sd->entity.num_pads ||
-		    !(sd->entity.pads[route->sink_pad].flags & MEDIA_PAD_FL_SINK)) {
+		    !(sd->entity.pads[route->sink_pad].flags &
+		      (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL_SOURCE))) {

So this relates to my reply to the previous patch: calling a sink pad an INTERNAL_SOURCE is really confusing.

Some suggestions for alternative names:

1) MEDIA_PAD_FL_SINK_INTERNAL
2) MEDIA_PAD_FL_SINK_INTERNAL_SOURCE (or _SRC)
3) MEDIA_PAD_FL_SINK_FROM_INTERNAL_SOURCE (or _SRC)

IMHO the name *has* to start with MEDIA_PAD_FL_SINK_ to indicate that it is a sink pad, followed by something that indicates that it is fed from an internal source.

Regards,

Hans

Quoted 3 lines
drivers/media/v4l2-core/v4l2-subdev.c +0 -0
 			dev_dbg(sd->dev, "route %u sink (%u) is not a sink pad\n",
 				i, route->sink_pad);
 			goto out;
author maintainer
#20 lore raw

RFC 3/7 media: uapi: v4l: Document source routes

In reply to Sakari Ailus

Document how internal pads are used on source routes.

---
 .../userspace-api/media/v4l/dev-subdev.rst     | 18 ++++++++++++++++++
 .../media/v4l/vidioc-subdev-g-routing.rst      |  5 +++++
 include/uapi/linux/v4l2-subdev.h               |  6 +++++-
 3 files changed, 28 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/v4l/dev-subdev.rst +18 -0
index a4f1df7093e8..395e06d2f0f2 100644
@@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the
 sub-device and a (pad, stream) pair. For sub-devices that do not support
 multiplexed streams the 'stream' field is always 0.
 
+.. _v4l2-subdev-source-routes:
+
+Source routes
+^^^^^^^^^^^^^
+
+Cases where a single sub-device pad is a source of multiple streams are special
+as there is no sink pad for such a route. In those cases, an internal pad is
+used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCE
+<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set.
+
+Internal pads have all the properties of a sink pad in such case, including
+formats and selections. The format in this case is the source format of the
+stream. An internal pad always has a single stream only (0).
+
+Generally source routes are not modifiable but they can be activated and
+inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
+<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
+
 Interaction between routes, streams, formats and selections
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst +5 -0
index 68ca343c3b44..e00622071b64 100644
@@ -94,6 +94,11 @@ for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
     * - __u32
       - ``sink_pad``
       - Sink pad number.
+    * - __u32
+      - ``internal_source_pad``
+      - Internal source pad number. For pads with :ref:`internal source pad flag
+	<MEDIA-PAD-FL-INTERNAL-SOURCE>`, for use with :ref:`source routes
+	<v4l2-subdev-source-routes>`.
     * - __u32
       - ``sink_stream``
       - Sink pad stream number.
include/uapi/linux/v4l2-subdev.h +5 -1
index 4a195b68f28f..703e3a1f199b 100644
@@ -203,6 +203,7 @@ struct v4l2_subdev_capability {
  * struct v4l2_subdev_route - A route inside a subdev
  *
  * @sink_pad: the sink pad index
+ * @internal_source_pad: the internal source pad
  * @sink_stream: the sink stream identifier
  * @source_pad: the source pad index
  * @source_stream: the source stream identifier
@@ -210,7 +211,10 @@ struct v4l2_subdev_capability {
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_route {
-	__u32 sink_pad;
+	union {
+		__u32 sink_pad;
+		__u32 internal_source_pad;
+	};
 	__u32 sink_stream;
 	__u32 source_pad;
 	__u32 source_stream;
Signature · 1 line
2.30.2
#21 lore raw

Re: [RFC 3/7] media: uapi: v4l: Document source routes

Quoted 25 lines · Sakari Ailus

Document how internal pads are used on source routes.

---
  .../userspace-api/media/v4l/dev-subdev.rst     | 18 ++++++++++++++++++
  .../media/v4l/vidioc-subdev-g-routing.rst      |  5 +++++
  include/uapi/linux/v4l2-subdev.h               |  6 +++++-
  3 files changed, 28 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/v4l/dev-subdev.rst +8 -0
index a4f1df7093e8..395e06d2f0f2 100644
@@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the
  sub-device and a (pad, stream) pair. For sub-devices that do not support
  multiplexed streams the 'stream' field is always 0.
 
+.. _v4l2-subdev-source-routes:
+
+Source routes
+^^^^^^^^^^^^^
+
+Cases where a single sub-device pad is a source of multiple streams are special
+as there is no sink pad for such a route. In those cases, an internal pad is
+used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCE

Documentation/userspace-api/media/v4l/dev-subdev.rst A stream at a specific point in the media pipeline is identified by the · line 571

Quoted 10 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +10 -0
+<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set.
+
+Internal pads have all the properties of a sink pad in such case, including
+formats and selections. The format in this case is the source format of the
+stream. An internal pad always has a single stream only (0).
+
+Generally source routes are not modifiable but they can be activated and
+inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
+<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
+

I find the above chapter a bit difficult to read, but I think we need to conclude on the internal-pad vs internal-source-pad discussion first. However, one point/question:

You write that there's only one stream in an internal pad. I wonder if that's a good or a necessary limitation... Do you see that allowing multiple streams brings extra complexity? It's still up to each driver to decide how many streams they support (most would just support a single one), so each driver can easily enforce that limit.

I'm fine with starting with only single-stream support, but if we later need to loosen that restriction, I wonder if it'll be difficult if we have specified that there can be only one stream. I.e. will the drivers and the userspace depend on that limit.

Tomi

author maintainer
#22 lore raw

Re: [RFC 3/7] media: uapi: v4l: Document source routes

Moi,

Quoted 40 lines · Tomi Valkeinen
Quoted 24 lines · Sakari Ailus

Document how internal pads are used on source routes.

---
  .../userspace-api/media/v4l/dev-subdev.rst     | 18 ++++++++++++++++++
  .../media/v4l/vidioc-subdev-g-routing.rst      |  5 +++++
  include/uapi/linux/v4l2-subdev.h               |  6 +++++-
  3 files changed, 28 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/v4l/dev-subdev.rst +8 -0
index a4f1df7093e8..395e06d2f0f2 100644
@@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the
  sub-device and a (pad, stream) pair. For sub-devices that do not support
  multiplexed streams the 'stream' field is always 0.
+.. _v4l2-subdev-source-routes:
+
+Source routes
+^^^^^^^^^^^^^
+
+Cases where a single sub-device pad is a source of multiple streams are special
+as there is no sink pad for such a route. In those cases, an internal pad is
+used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCE

Documentation/userspace-api/media/v4l/dev-subdev.rst A stream at a specific point in the media pipeline is identified by the · line 570

Quoted 10 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +10 -0
+<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set.
+
+Internal pads have all the properties of a sink pad in such case, including
+formats and selections. The format in this case is the source format of the
+stream. An internal pad always has a single stream only (0).
+
+Generally source routes are not modifiable but they can be activated and
+inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
+<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
+

I find the above chapter a bit difficult to read, but I think we need to conclude on the internal-pad vs internal-source-pad discussion first. However, one point/question:

Quoted 5 lines

You write that there's only one stream in an internal pad. I wonder if that's a good or a necessary limitation... Do you see that allowing multiple streams brings extra complexity? It's still up to each driver to decide how many streams they support (most would just support a single one), so each driver can easily enforce that limit.

Good question. As we don't seem to have a tangible reason to allow it, I'd deny it until there's a use case.

Or did you have a use case in mind?

I thought indicating which streams will be bound together (i.e. either are all disabled or enabled) could be one, but I'm not sure we need that at the moment at least.

Quoted 5 lines

I'm fine with starting with only single-stream support, but if we later need to loosen that restriction, I wonder if it'll be difficult if we have specified that there can be only one stream. I.e. will the drivers and the userspace depend on that limit.

We can always allow what wasn't allowed previously so that's a non-issue, really. But it needs to bring new functionality, otherwise there's no reason to do that.

Signature · 3 lines
Kind regards,

Sakari Ailus
#23 lore raw

Re: [RFC 3/7] media: uapi: v4l: Document source routes

Quoted 49 lines · Sakari Ailus

Moi,

Quoted 40 lines · Tomi Valkeinen
Quoted 24 lines · Sakari Ailus

Document how internal pads are used on source routes.

---
   .../userspace-api/media/v4l/dev-subdev.rst     | 18 ++++++++++++++++++
   .../media/v4l/vidioc-subdev-g-routing.rst      |  5 +++++
   include/uapi/linux/v4l2-subdev.h               |  6 +++++-
   3 files changed, 28 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/v4l/dev-subdev.rst +8 -0
index a4f1df7093e8..395e06d2f0f2 100644
@@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the
   sub-device and a (pad, stream) pair. For sub-devices that do not support
   multiplexed streams the 'stream' field is always 0.
+.. _v4l2-subdev-source-routes:
+
+Source routes
+^^^^^^^^^^^^^
+
+Cases where a single sub-device pad is a source of multiple streams are special
+as there is no sink pad for such a route. In those cases, an internal pad is
+used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCE

Documentation/userspace-api/media/v4l/dev-subdev.rst A stream at a specific point in the media pipeline is identified by the · line 570

Quoted 10 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +10 -0
+<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set.
+
+Internal pads have all the properties of a sink pad in such case, including
+formats and selections. The format in this case is the source format of the
+stream. An internal pad always has a single stream only (0).
+
+Generally source routes are not modifiable but they can be activated and
+inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
+<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
+

I find the above chapter a bit difficult to read, but I think we need to conclude on the internal-pad vs internal-source-pad discussion first. However, one point/question:

Quoted 5 lines

You write that there's only one stream in an internal pad. I wonder if that's a good or a necessary limitation... Do you see that allowing multiple streams brings extra complexity? It's still up to each driver to decide how many streams they support (most would just support a single one), so each driver can easily enforce that limit.

Quoted 8 lines

Good question. As we don't seem to have a tangible reason to allow it, I'd deny it until there's a use case.

Or did you have a use case in mind?

I thought indicating which streams will be bound together (i.e. either are all disabled or enabled) could be one, but I'm not sure we need that at the moment at least.

I don't have nothing concrete in mind. Maybe a TPG which also generates some kind of metadata. But that could be implemented as two internal pads.

Quoted 8 lines
Quoted 4 lines

I'm fine with starting with only single-stream support, but if we later need to loosen that restriction, I wonder if it'll be difficult if we have specified that there can be only one stream. I.e. will the drivers and the userspace depend on that limit.

We can always allow what wasn't allowed previously so that's a non-issue, really. But it needs to bring new functionality, otherwise there's no reason to do that.

It's not always that easy. If the drivers and the userspace expect that there's a single route with ID 0, and then with a new kernel there are more streams or the single stream is ID 1, things could go wrong.

Tomi

author maintainer
#24 lore raw

Re: [RFC 3/7] media: uapi: v4l: Document source routes

Moi,

Quoted 67 lines · Tomi Valkeinen
Quoted 49 lines · Sakari Ailus

Moi,

Quoted 40 lines · Tomi Valkeinen
Quoted 24 lines · Sakari Ailus

Document how internal pads are used on source routes.

---
   .../userspace-api/media/v4l/dev-subdev.rst     | 18 ++++++++++++++++++
   .../media/v4l/vidioc-subdev-g-routing.rst      |  5 +++++
   include/uapi/linux/v4l2-subdev.h               |  6 +++++-
   3 files changed, 28 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/v4l/dev-subdev.rst +8 -0
index a4f1df7093e8..395e06d2f0f2 100644
@@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the
   sub-device and a (pad, stream) pair. For sub-devices that do not support
   multiplexed streams the 'stream' field is always 0.
+.. _v4l2-subdev-source-routes:
+
+Source routes
+^^^^^^^^^^^^^
+
+Cases where a single sub-device pad is a source of multiple streams are special
+as there is no sink pad for such a route. In those cases, an internal pad is
+used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCE

Documentation/userspace-api/media/v4l/dev-subdev.rst A stream at a specific point in the media pipeline is identified by the · line 570

Quoted 10 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +10 -0
+<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set.
+
+Internal pads have all the properties of a sink pad in such case, including
+formats and selections. The format in this case is the source format of the
+stream. An internal pad always has a single stream only (0).
+
+Generally source routes are not modifiable but they can be activated and
+inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
+<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
+

I find the above chapter a bit difficult to read, but I think we need to conclude on the internal-pad vs internal-source-pad discussion first. However, one point/question:

Quoted 5 lines

You write that there's only one stream in an internal pad. I wonder if that's a good or a necessary limitation... Do you see that allowing multiple streams brings extra complexity? It's still up to each driver to decide how many streams they support (most would just support a single one), so each driver can easily enforce that limit.

Quoted 8 lines

Good question. As we don't seem to have a tangible reason to allow it, I'd deny it until there's a use case.

Or did you have a use case in mind?

I thought indicating which streams will be bound together (i.e. either are all disabled or enabled) could be one, but I'm not sure we need that at the moment at least.

I don't have nothing concrete in mind. Maybe a TPG which also generates some kind of metadata. But that could be implemented as two internal pads.

Quoted 5 lines
Quoted 4 lines

I'm fine with starting with only single-stream support, but if we later need to loosen that restriction, I wonder if it'll be difficult if we have specified that there can be only one stream. I.e. will the drivers and the userspace depend on that limit.

Quoted 7 lines
Quoted 3 lines

We can always allow what wasn't allowed previously so that's a non-issue, really. But it needs to bring new functionality, otherwise there's no reason to do that.

It's not always that easy. If the drivers and the userspace expect that there's a single route with ID 0, and then with a new kernel there are more streams or the single stream is ID 1, things could go wrong.

Other drivers are a non-issue IMO as this isn't visible to other drivers.

I think for user space this could be an issue if it were entirely generic. But that's probably not going to be the case, is it? I don't mean test programs.

Signature · 3 lines
Terveisin,

Sakari Ailus
maintainer
#25 lore raw

Re: [RFC 3/7] media: uapi: v4l: Document source routes

Quoted 74 lines · Sakari Ailus
Quoted 65 lines · Tomi Valkeinen
Quoted 47 lines · Sakari Ailus
Quoted 40 lines · Tomi Valkeinen
Quoted 24 lines · Sakari Ailus

Document how internal pads are used on source routes.

---
   .../userspace-api/media/v4l/dev-subdev.rst     | 18 ++++++++++++++++++
   .../media/v4l/vidioc-subdev-g-routing.rst      |  5 +++++
   include/uapi/linux/v4l2-subdev.h               |  6 +++++-
   3 files changed, 28 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/v4l/dev-subdev.rst +8 -0
index a4f1df7093e8..395e06d2f0f2 100644
@@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the
   sub-device and a (pad, stream) pair. For sub-devices that do not support
   multiplexed streams the 'stream' field is always 0.
+.. _v4l2-subdev-source-routes:
+
+Source routes
+^^^^^^^^^^^^^
+
+Cases where a single sub-device pad is a source of multiple streams are special
+as there is no sink pad for such a route. In those cases, an internal pad is
+used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCE

Documentation/userspace-api/media/v4l/dev-subdev.rst A stream at a specific point in the media pipeline is identified by the · line 570

Quoted 10 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +10 -0
+<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set.
+
+Internal pads have all the properties of a sink pad in such case, including
+formats and selections. The format in this case is the source format of the
+stream. An internal pad always has a single stream only (0).
+
+Generally source routes are not modifiable but they can be activated and
+inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
+<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
+

I find the above chapter a bit difficult to read, but I think we need to conclude on the internal-pad vs internal-source-pad discussion first. However, one point/question:

Quoted 5 lines

You write that there's only one stream in an internal pad. I wonder if that's a good or a necessary limitation... Do you see that allowing multiple streams brings extra complexity? It's still up to each driver to decide how many streams they support (most would just support a single one), so each driver can easily enforce that limit.

Quoted 8 lines

Good question. As we don't seem to have a tangible reason to allow it, I'd deny it until there's a use case.

Or did you have a use case in mind?

I thought indicating which streams will be bound together (i.e. either are all disabled or enabled) could be one, but I'm not sure we need that at the moment at least.

I don't have nothing concrete in mind. Maybe a TPG which also generates some kind of metadata. But that could be implemented as two internal pads.

Quoted 5 lines
Quoted 4 lines

I'm fine with starting with only single-stream support, but if we later need to loosen that restriction, I wonder if it'll be difficult if we have specified that there can be only one stream. I.e. will the drivers and the userspace depend on that limit.

Quoted 7 lines
Quoted 3 lines

We can always allow what wasn't allowed previously so that's a non-issue, really. But it needs to bring new functionality, otherwise there's no reason to do that.

It's not always that easy. If the drivers and the userspace expect that there's a single route with ID 0, and then with a new kernel there are more streams or the single stream is ID 1, things could go wrong.

Quoted 5 lines

Other drivers are a non-issue IMO as this isn't visible to other drivers.

I think for user space this could be an issue if it were entirely generic. But that's probably not going to be the case, is it? I don't mean test programs.

I could imagine this breaking libcamera for instance.

Signature · 3 lines
Regards,

Laurent Pinchart
maintainer
#26 lore raw

Re: [RFC 3/7] media: uapi: v4l: Document source routes

In reply to Tomi Valkeinen

Hello,

Quoted 33 lines · Tomi Valkeinen
Quoted 32 lines · Sakari Ailus
Quoted 31 lines · Tomi Valkeinen
Quoted 30 lines · Sakari Ailus

Document how internal pads are used on source routes.

---
   .../userspace-api/media/v4l/dev-subdev.rst     | 18 ++++++++++++++++++
   .../media/v4l/vidioc-subdev-g-routing.rst      |  5 +++++
   include/uapi/linux/v4l2-subdev.h               |  6 +++++-
   3 files changed, 28 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/v4l/dev-subdev.rst +14 -0
index a4f1df7093e8..395e06d2f0f2 100644
@@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the
   sub-device and a (pad, stream) pair. For sub-devices that do not support
   multiplexed streams the 'stream' field is always 0.
+.. _v4l2-subdev-source-routes:
+
+Source routes
+^^^^^^^^^^^^^
+
+Cases where a single sub-device pad is a source of multiple streams are special
+as there is no sink pad for such a route. In those cases, an internal pad is
+used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCE
+<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set.
+
+Internal pads have all the properties of a sink pad in such case, including
+formats and selections. The format in this case is the source format of the
+stream. An internal pad always has a single stream only (0).
+
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Documentation/userspace-api/media/v4l/dev-subdev.rst A stream at a specific point in the media pipeline is identified by the · line 571

Quoted 4 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +4 -0
+Generally source routes are not modifiable but they can be activated and
+inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
+<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
+

I find the above chapter a bit difficult to read, but I think we need to conclude on the internal-pad vs internal-source-pad discussion first. However, one point/question:

Agreed, it's far from being clear :-( Even the first sentence, "Cases where a single sub-device pad is a source of multiple streams are special" is confusing, having a subdev source pad carrying multiple streams isn't special, it is found in, for instance, FPD-Link or GMSL receivers that transmit multiple streams from different cameras over a single CSI-2 output. This may not be what Sakari meant here, but it can be understood that way.

We need more than 3 paragraphs here, and we need a very clear example with a diagram. I'd recommend using a camera sensor that produces image data and embedded data for the example, as that's a common real-life use case. The text should present the example, explain what the problem is, and then explain how internal pads fix it. Then it can expand on the features and usage of internal pads in a more generic way.

Quoted 23 lines
Quoted 6 lines
Quoted 5 lines

You write that there's only one stream in an internal pad. I wonder if that's a good or a necessary limitation... Do you see that allowing multiple streams brings extra complexity? It's still up to each driver to decide how many streams they support (most would just support a single one), so each driver can easily enforce that limit.

Quoted 8 lines

Good question. As we don't seem to have a tangible reason to allow it, I'd deny it until there's a use case.

Or did you have a use case in mind?

I thought indicating which streams will be bound together (i.e. either are all disabled or enabled) could be one, but I'm not sure we need that at the moment at least.

I don't have nothing concrete in mind. Maybe a TPG which also generates some kind of metadata. But that could be implemented as two internal pads.

Quoted 5 lines
Quoted 4 lines

I'm fine with starting with only single-stream support, but if we later need to loosen that restriction, I wonder if it'll be difficult if we have specified that there can be only one stream. I.e. will the drivers and the userspace depend on that limit.

Quoted 7 lines
Quoted 3 lines

We can always allow what wasn't allowed previously so that's a non-issue, really. But it needs to bring new functionality, otherwise there's no reason to do that.

It's not always that easy. If the drivers and the userspace expect that there's a single route with ID 0, and then with a new kernel there are more streams or the single stream is ID 1, things could go wrong.

I agree with Tomi here. On the kernel side it should be fine (with an unknown amount of pain), but I'd consider this as a userspace API breakage. If the specifications indicates that only a single stream can be used, applications may rely on that, and things could go wrong if new streams are added.

We can restrict the kernel implementation to a single stream, but the userspace API has to indicate that multiple streams can co-exist if we want to allow that in the future.

Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#27 lore raw

Re: [RFC 3/7] media: uapi: v4l: Document source routes

Hi Laurent,

Quoted 45 lines · Laurent Pinchart

Hello,

Quoted 33 lines · Tomi Valkeinen
Quoted 32 lines · Sakari Ailus
Quoted 31 lines · Tomi Valkeinen
Quoted 30 lines · Sakari Ailus

Document how internal pads are used on source routes.

---
   .../userspace-api/media/v4l/dev-subdev.rst     | 18 ++++++++++++++++++
   .../media/v4l/vidioc-subdev-g-routing.rst      |  5 +++++
   include/uapi/linux/v4l2-subdev.h               |  6 +++++-
   3 files changed, 28 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/v4l/dev-subdev.rst +14 -0
index a4f1df7093e8..395e06d2f0f2 100644
@@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the
   sub-device and a (pad, stream) pair. For sub-devices that do not support
   multiplexed streams the 'stream' field is always 0.
+.. _v4l2-subdev-source-routes:
+
+Source routes
+^^^^^^^^^^^^^
+
+Cases where a single sub-device pad is a source of multiple streams are special
+as there is no sink pad for such a route. In those cases, an internal pad is
+used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCE
+<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set.
+
+Internal pads have all the properties of a sink pad in such case, including
+formats and selections. The format in this case is the source format of the
+stream. An internal pad always has a single stream only (0).
+
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Documentation/userspace-api/media/v4l/dev-subdev.rst A stream at a specific point in the media pipeline is identified by the · line 571

Quoted 4 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +4 -0
+Generally source routes are not modifiable but they can be activated and
+inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
+<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
+

I find the above chapter a bit difficult to read, but I think we need to conclude on the internal-pad vs internal-source-pad discussion first. However, one point/question:

Quoted 7 lines

Agreed, it's far from being clear :-( Even the first sentence, "Cases where a single sub-device pad is a source of multiple streams are special" is confusing, having a subdev source pad carrying multiple streams isn't special, it is found in, for instance, FPD-Link or GMSL receivers that transmit multiple streams from different cameras over a single CSI-2 output. This may not be what Sakari meant here, but it can be understood that way.

The sentence continues on the next line. I'll rework this part of the documentation in any case while moving to a plain INTERNAL flag.

Quoted 7 lines

We need more than 3 paragraphs here, and we need a very clear example with a diagram. I'd recommend using a camera sensor that produces image data and embedded data for the example, as that's a common real-life use case. The text should present the example, explain what the problem is, and then explain how internal pads fix it. Then it can expand on the features and usage of internal pads in a more generic way.

It's fine to have examples but I'd add them to where the other examples are located.

Quoted 38 lines
Quoted 24 lines
Quoted 7 lines
Quoted 6 lines

You write that there's only one stream in an internal pad. I wonder if that's a good or a necessary limitation... Do you see that allowing multiple streams brings extra complexity? It's still up to each driver to decide how many streams they support (most would just support a single one), so each driver can easily enforce that limit.

Quoted 8 lines

Good question. As we don't seem to have a tangible reason to allow it, I'd deny it until there's a use case.

Or did you have a use case in mind?

I thought indicating which streams will be bound together (i.e. either are all disabled or enabled) could be one, but I'm not sure we need that at the moment at least.

I don't have nothing concrete in mind. Maybe a TPG which also generates some kind of metadata. But that could be implemented as two internal pads.

Quoted 5 lines
Quoted 4 lines

I'm fine with starting with only single-stream support, but if we later need to loosen that restriction, I wonder if it'll be difficult if we have specified that there can be only one stream. I.e. will the drivers and the userspace depend on that limit.

Quoted 7 lines
Quoted 3 lines

We can always allow what wasn't allowed previously so that's a non-issue, really. But it needs to bring new functionality, otherwise there's no reason to do that.

It's not always that easy. If the drivers and the userspace expect that there's a single route with ID 0, and then with a new kernel there are more streams or the single stream is ID 1, things could go wrong.

I agree with Tomi here. On the kernel side it should be fine (with an unknown amount of pain), but I'd consider this as a userspace API breakage. If the specifications indicates that only a single stream can be used, applications may rely on that, and things could go wrong if new streams are added.

Quoted 3 lines

We can restrict the kernel implementation to a single stream, but the userspace API has to indicate that multiple streams can co-exist if we want to allow that in the future.

I doubt this ever could be an actual problem. Allowing multiple streams to be merged on a source pad requires the user to first configure them.

I find it hard to believe there would be an application that tried to configure routes this way (which would have always failed) and then proceeded anyway, and somehow returning success from route setup leads to routing configuration that the application couldn't handle anymore.

Signature · 3 lines
Kind regards,

Sakari Ailus
#28 lore raw

Re: [RFC 3/7] media: uapi: v4l: Document source routes

In reply to Sakari Ailus
Quoted 16 lines · Sakari Ailus

Document how internal pads are used on source routes.

---
 .../userspace-api/media/v4l/dev-subdev.rst     | 18 ++++++++++++++++++
 .../media/v4l/vidioc-subdev-g-routing.rst      |  5 +++++
 include/uapi/linux/v4l2-subdev.h               |  6 +++++-
 3 files changed, 28 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/v4l/dev-subdev.rst +0 -0
index a4f1df7093e8..395e06d2f0f2 100644
@@ -551,6 +551,24 @@ A stream at a specific point in the media pipeline is identified by the
 sub-device and a (pad, stream) pair. For sub-devices that do not support
 multiplexed streams the 'stream' field is always 0.

Documentation/userspace-api/media/v4l/dev-subdev.rst A stream at a specific point in the media pipeline is identified by the · line 562

Quoted 10 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +9 -0
 
+.. _v4l2-subdev-source-routes:
+
+Source routes
+^^^^^^^^^^^^^
+
+Cases where a single sub-device pad is a source of multiple streams are special
+as there is no sink pad for such a route. In those cases, an internal pad is
+used as the sink pad. Such pads have the :ref:`MEDIA_PAD_FL_INTERNAL_SOURCE
+<MEDIA-PAD-FL-INTERNAL-SOURCE>` flag set.

add: instead of the <MEDIA-PAD-FL-SINK> flag.

Documentation/userspace-api/media/v4l/dev-subdev.rst A stream at a specific point in the media pipeline is identified by the · line 569

Quoted 7 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +7 -0
+
+Internal pads have all the properties of a sink pad in such case, including
+formats and selections. The format in this case is the source format of the
+stream. An internal pad always has a single stream only (0).
+
+Generally source routes are not modifiable but they can be activated and
+inactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE

inactivated -> deactivated

Documentation/userspace-api/media/v4l/dev-subdev.rst A stream at a specific point in the media pipeline is identified by the · line 571

Quoted 2 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +2 -0
+<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
+

Again, this really needs one or more examples.

Quoted 32 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +0 -0
 Interaction between routes, streams, formats and selections
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst +5 -0
index 68ca343c3b44..e00622071b64 100644
@@ -94,6 +94,11 @@ for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
     * - __u32
       - ``sink_pad``
       - Sink pad number.
+    * - __u32
+      - ``internal_source_pad``
+      - Internal source pad number. For pads with :ref:`internal source pad flag
+	<MEDIA-PAD-FL-INTERNAL-SOURCE>`, for use with :ref:`source routes
+	<v4l2-subdev-source-routes>`.
     * - __u32
       - ``sink_stream``
       - Sink pad stream number.
include/uapi/linux/v4l2-subdev.h +1 -0
index 4a195b68f28f..703e3a1f199b 100644
@@ -203,6 +203,7 @@ struct v4l2_subdev_capability {
  * struct v4l2_subdev_route - A route inside a subdev
  *
  * @sink_pad: the sink pad index
+ * @internal_source_pad: the internal source pad
  * @sink_stream: the sink stream identifier
  * @source_pad: the source pad index
  * @source_stream: the source stream identifier

include/uapi/linux/v4l2-subdev.h struct v4l2_subdev_capability { · line 216

Quoted 7 lines
include/uapi/linux/v4l2-subdev.h +3 -1
@@ -210,7 +211,10 @@ struct v4l2_subdev_capability {
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_route {
-	__u32 sink_pad;
+	union {
+		__u32 sink_pad;
+		__u32 internal_source_pad;

Nobody will understand this. I think this should still be kept as sink_pad. If the INTERNAL_SOURCE flag is changed to SINK_FROM_INTERNAL_SOURCE (or something along those lines), then 'sink_pad' still makes sense and you don't need this change.

It's still a sink pad, just a sink pad with a special property in that it receives the data from an internal source.

include/uapi/linux/v4l2-subdev.h struct v4l2_subdev_capability { · line 220

Quoted 4 lines
include/uapi/linux/v4l2-subdev.h +1 -0
+	};
 	__u32 sink_stream;
 	__u32 source_pad;
 	__u32 source_stream;

Regards,

Hans

author maintainer
#29 lore raw

RFC 4/7 media: mc: Check pad flag validity

In reply to Sakari Ailus

Check the validity of pad flags on entity init. Exactly one of the flags must be set.

---
 drivers/media/mc/mc-entity.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
drivers/media/mc/mc-entity.c +11 -1
index ed99160a2487..c1bc48c4d239 100644
@@ -197,6 +197,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	struct media_device *mdev = entity->graph_obj.mdev;
 	struct media_pad *iter;
 	unsigned int i = 0;
+	int ret = 0;
 
 	if (num_pads >= MEDIA_ENTITY_MAX_PADS)
 		return -E2BIG;
@@ -210,6 +211,15 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	media_entity_for_each_pad(entity, iter) {
 		iter->entity = entity;
 		iter->index = i++;
+
+		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
+					     MEDIA_PAD_FL_SOURCE |
+					     MEDIA_PAD_FL_INTERNAL_SOURCE))
+		    != 1) {
+			ret = -EINVAL;
+			break;
+		}
+
 		if (mdev)
 			media_gobj_create(mdev, MEDIA_GRAPH_PAD,
 					  &iter->graph_obj);
@@ -218,7 +228,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	if (mdev)
 		mutex_unlock(&mdev->graph_mutex);
 
-	return 0;
+	return ret;
 }
 EXPORT_SYMBOL_GPL(media_entity_pads_init);
 
Signature · 1 line
2.30.2
maintainer
#30 lore raw

Re: [RFC 4/7] media: mc: Check pad flag validity

Hi Sakari,

Thank you for the patch.

Quoted 21 lines · Sakari Ailus

Check the validity of pad flags on entity init. Exactly one of the flags must be set.

---
 drivers/media/mc/mc-entity.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
drivers/media/mc/mc-entity.c +1 -0
index ed99160a2487..c1bc48c4d239 100644
@@ -197,6 +197,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	struct media_device *mdev = entity->graph_obj.mdev;
 	struct media_pad *iter;
 	unsigned int i = 0;
+	int ret = 0;
 
 	if (num_pads >= MEDIA_ENTITY_MAX_PADS)
 		return -E2BIG;

drivers/media/mc/mc-entity.c int media_entity_pads_init(struct media_entity *entity, u16 num_pads, · line 218

Quoted 8 lines
drivers/media/mc/mc-entity.c +5 -0
@@ -210,6 +211,15 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	media_entity_for_each_pad(entity, iter) {
 		iter->entity = entity;
 		iter->index = i++;
+
+		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
+					     MEDIA_PAD_FL_SOURCE |
+					     MEDIA_PAD_FL_INTERNAL_SOURCE))
+		    != 1) {

I would add this check as patch 1/7 in this series, with FL_SINK an FL_SOURCE only, and then extend it in the patch where you introduce FL_INTERNAL_SOURCE.

Quoted 16 lines
drivers/media/mc/mc-entity.c +5 -1
+			ret = -EINVAL;
+			break;
+		}
+
 		if (mdev)
 			media_gobj_create(mdev, MEDIA_GRAPH_PAD,
 					  &iter->graph_obj);
@@ -218,7 +228,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	if (mdev)
 		mutex_unlock(&mdev->graph_mutex);
 
-	return 0;
+	return ret;
 }
 EXPORT_SYMBOL_GPL(media_entity_pads_init);
 
Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#31 lore raw

Re: [RFC 4/7] media: mc: Check pad flag validity

Quoted 23 lines · Laurent Pinchart

Hi Sakari,

Thank you for the patch.

Quoted 18 lines · Sakari Ailus

Check the validity of pad flags on entity init. Exactly one of the flags must be set.

---
 drivers/media/mc/mc-entity.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
drivers/media/mc/mc-entity.c +1 -0
index ed99160a2487..c1bc48c4d239 100644
@@ -197,6 +197,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	struct media_device *mdev = entity->graph_obj.mdev;
 	struct media_pad *iter;
 	unsigned int i = 0;
+	int ret = 0;
 
Quoted 15 lines

drivers/media/mc/mc-entity.c int media_entity_pads_init(struct media_entity *entity, u16 num_pads, · line 218

Quoted 11 lines
drivers/media/mc/mc-entity.c +5 -0
 	if (num_pads >= MEDIA_ENTITY_MAX_PADS)
 		return -E2BIG;
@@ -210,6 +211,15 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	media_entity_for_each_pad(entity, iter) {
 		iter->entity = entity;
 		iter->index = i++;
+
+		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
+					     MEDIA_PAD_FL_SOURCE |
+					     MEDIA_PAD_FL_INTERNAL_SOURCE))
+		    != 1) {

I would add this check as patch 1/7 in this series, with FL_SINK an FL_SOURCE only, and then extend it in the patch where you introduce FL_INTERNAL_SOURCE.

Sounds reasonable.

Quoted 22 lines
Quoted 17 lines
drivers/media/mc/mc-entity.c +5 -1
+			ret = -EINVAL;
+			break;
+		}
+
 		if (mdev)
 			media_gobj_create(mdev, MEDIA_GRAPH_PAD,
 					  &iter->graph_obj);
@@ -218,7 +228,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	if (mdev)
 		mutex_unlock(&mdev->graph_mutex);
 
-	return 0;
+	return ret;
 }
 EXPORT_SYMBOL_GPL(media_entity_pads_init);
 
Signature · 3 lines
Regards,

Laurent Pinchart
Signature · 1 line
Sakari Ailus
author maintainer
#32 lore raw

RFC 5/7 media: uapi: Add generic serial metadata mbus formats

In reply to Sakari Ailus

Add generic serial metadata mbus formats. These formats describe data width and packing but not the content itself. The reason for specifying such formats is that the formats as such are fairly device specific but they are still handled by CSI-2 receiver drivers that should not be aware of device specific formats. What makes generic metadata formats possible is that these formats are parsed by software only, after capturing the data to system memory.

---
 .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++++++
 include/uapi/linux/media-bus-format.h         |   9 +
 2 files changed, 266 insertions(+)
Documentation/userspace-api/media/v4l/subdev-formats.rst +257 -0
index a3a35eeed708..1492fff58426 100644
@@ -8234,3 +8234,260 @@ The following table lists the existing metadata formats.
 	both sides of the link and the bus format is a fixed
 	metadata format that is not configurable from userspace.
 	Width and height will be set to 0 for this format.
+
+Generic Serial Metadata Formats
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Generic serial metadata formats are used on serial busses where the actual data
+content is more or less device specific but the data is transmitted and received
+by multiple devices that do not process the data in any way, simply writing
+it to system memory for processing in software at the end of the pipeline.
+
+The more specific variant describing the actual data is used on the internal
+source pad of the originating sub-device.
+
+"b" in an array cell signifies a byte of data, followed by the number of byte
+and finally the bit number in subscript. "p" indicates a padding bit.
+
+.. _media-bus-format-generic-meta:
+
+.. cssclass: longtable
+
+.. flat-table:: Generic Serial Metadata Formats
+    :header-rows:  2
+    :stub-columns: 0
+
+    * - Identifier
+      - Code
+      -
+      - :cspan:`23` Data organization
+    * -
+      -
+      - Bit
+      - 23
+      - 22
+      - 21
+      - 20
+      - 19
+      - 18
+      - 17
+      - 16
+      - 15
+      - 14
+      - 13
+      - 12
+      - 11
+      - 10
+      - 9
+      - 8
+      - 7
+      - 6
+      - 5
+      - 4
+      - 3
+      - 2
+      - 1
+      - 0
+    * .. _MEDIA-BUS-FMT-META-1X8-8:
+
+      - MEDIA_BUS_FMT_META_1X8_8
+      - 0x8001
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+    * .. _MEDIA-BUS-FMT-META-1X8-10:
+
+      - MEDIA_BUS_FMT_META_1X8_10
+      - 0x8002
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-12:
+
+      - MEDIA_BUS_FMT_META_1X8_12
+      - 0x8003
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-14:
+
+      - MEDIA_BUS_FMT_META_1X8_14
+      - 0x8004
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-16:
+
+      - MEDIA_BUS_FMT_META_1X8_16
+      - 0x8005
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-20:
+
+      - MEDIA_BUS_FMT_META_1X8_20
+      - 0x8007
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-24:
+
+      - MEDIA_BUS_FMT_META_1X8_24
+      - 0x8009
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
include/uapi/linux/media-bus-format.h +9 -0
index a03c543cb072..722463523bbd 100644
@@ -173,4 +173,13 @@
  */
 #define MEDIA_BUS_FMT_METADATA_FIXED		0x7001
 
+/* Generic line based metadata formats for serial buses. Next is 0x800b. */
+#define MEDIA_BUS_FMT_META_1X8_8		0x8001
+#define MEDIA_BUS_FMT_META_1X8_10		0x8002
+#define MEDIA_BUS_FMT_META_1X8_12		0x8003
+#define MEDIA_BUS_FMT_META_1X8_14		0x8004
+#define MEDIA_BUS_FMT_META_1X8_16		0x8005
+#define MEDIA_BUS_FMT_META_1X8_20		0x8007
+#define MEDIA_BUS_FMT_META_1X8_24		0x8009
+
 #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
Signature · 1 line
2.30.2
maintainer
#33 lore raw

Re: [RFC 5/7] media: uapi: Add generic serial metadata mbus formats

Hi Sakari,

Thank you for the patch.

Quoted 19 lines · Sakari Ailus

Add generic serial metadata mbus formats. These formats describe data width and packing but not the content itself. The reason for specifying such formats is that the formats as such are fairly device specific but they are still handled by CSI-2 receiver drivers that should not be aware of device specific formats. What makes generic metadata formats possible is that these formats are parsed by software only, after capturing the data to system memory.

---
 .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++++++
 include/uapi/linux/media-bus-format.h         |   9 +
 2 files changed, 266 insertions(+)
Documentation/userspace-api/media/v4l/subdev-formats.rst +0 -0
index a3a35eeed708..1492fff58426 100644

Documentation/userspace-api/media/v4l/subdev-formats.rst The following table lists the existing metadata formats. · line 8241

Quoted 8 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +5 -0
@@ -8234,3 +8234,260 @@ The following table lists the existing metadata formats.
 	both sides of the link and the bus format is a fixed
 	metadata format that is not configurable from userspace.
 	Width and height will be set to 0 for this format.
+
+Generic Serial Metadata Formats
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Generic serial metadata formats are used on serial busses where the actual data

s/busses/buses/

Documentation/userspace-api/media/v4l/subdev-formats.rst The following table lists the existing metadata formats. · line 8247

Quoted 6 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +6 -0
+content is more or less device specific but the data is transmitted and received
+by multiple devices that do not process the data in any way, simply writing
+it to system memory for processing in software at the end of the pipeline.
+
+The more specific variant describing the actual data is used on the internal
+source pad of the originating sub-device.

What do you mean by "more specific variant" here ? Please include an example in the documentation.

I'm not sure I like mentioning internal source pads here, are we guaranteed that metadata will always originate from an internal source pad ?

Quoted 252 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +246 -0
+
+"b" in an array cell signifies a byte of data, followed by the number of byte
+and finally the bit number in subscript. "p" indicates a padding bit.
+
+.. _media-bus-format-generic-meta:
+
+.. cssclass: longtable
+
+.. flat-table:: Generic Serial Metadata Formats
+    :header-rows:  2
+    :stub-columns: 0
+
+    * - Identifier
+      - Code
+      -
+      - :cspan:`23` Data organization
+    * -
+      -
+      - Bit
+      - 23
+      - 22
+      - 21
+      - 20
+      - 19
+      - 18
+      - 17
+      - 16
+      - 15
+      - 14
+      - 13
+      - 12
+      - 11
+      - 10
+      - 9
+      - 8
+      - 7
+      - 6
+      - 5
+      - 4
+      - 3
+      - 2
+      - 1
+      - 0
+    * .. _MEDIA-BUS-FMT-META-1X8-8:
+
+      - MEDIA_BUS_FMT_META_1X8_8
+      - 0x8001
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+    * .. _MEDIA-BUS-FMT-META-1X8-10:
+
+      - MEDIA_BUS_FMT_META_1X8_10
+      - 0x8002
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-12:
+
+      - MEDIA_BUS_FMT_META_1X8_12
+      - 0x8003
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-14:
+
+      - MEDIA_BUS_FMT_META_1X8_14
+      - 0x8004
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-16:
+
+      - MEDIA_BUS_FMT_META_1X8_16
+      - 0x8005
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-20:
+
+      - MEDIA_BUS_FMT_META_1X8_20
+      - 0x8007
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-24:
+
+      - MEDIA_BUS_FMT_META_1X8_24
+      - 0x8009
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
include/uapi/linux/media-bus-format.h +0 -0
index a03c543cb072..722463523bbd 100644
@@ -173,4 +173,13 @@
  */

include/uapi/linux/media-bus-format.h line 183

Quoted 10 lines
include/uapi/linux/media-bus-format.h +8 -0
 #define MEDIA_BUS_FMT_METADATA_FIXED		0x7001
 
+/* Generic line based metadata formats for serial buses. Next is 0x800b. */
+#define MEDIA_BUS_FMT_META_1X8_8		0x8001
+#define MEDIA_BUS_FMT_META_1X8_10		0x8002
+#define MEDIA_BUS_FMT_META_1X8_12		0x8003
+#define MEDIA_BUS_FMT_META_1X8_14		0x8004
+#define MEDIA_BUS_FMT_META_1X8_16		0x8005
+#define MEDIA_BUS_FMT_META_1X8_20		0x8007
+#define MEDIA_BUS_FMT_META_1X8_24		0x8009

We've discussed this before privately, it's time to come to a conclusion :-)

My preference would be

#define MEDIA_BUS_FMT_META_RAW8 0x8001
#define MEDIA_BUS_FMT_META_RAW10 0x8002
#define MEDIA_BUS_FMT_META_RAW12 0x8003
#define MEDIA_BUS_FMT_META_RAW14 0x8004
#define MEDIA_BUS_FMT_META_RAW16 0x8005
#define MEDIA_BUS_FMT_META_RAW20 0x8007
#define MEDIA_BUS_FMT_META_RAW24 0x8009

without defining the contents of the data (that is, no mention of padding bits).

Quoted 2 lines
include/uapi/linux/media-bus-format.h +1 -0
+
 #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#34 lore raw

Re: [RFC 5/7] media: uapi: Add generic serial metadata mbus formats

Hi Laurent,

Quoted 45 lines · Laurent Pinchart

Hi Sakari,

Thank you for the patch.

Quoted 19 lines · Sakari Ailus

Add generic serial metadata mbus formats. These formats describe data width and packing but not the content itself. The reason for specifying such formats is that the formats as such are fairly device specific but they are still handled by CSI-2 receiver drivers that should not be aware of device specific formats. What makes generic metadata formats possible is that these formats are parsed by software only, after capturing the data to system memory.

---
 .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++++++
 include/uapi/linux/media-bus-format.h         |   9 +
 2 files changed, 266 insertions(+)
Documentation/userspace-api/media/v4l/subdev-formats.rst +0 -0
index a3a35eeed708..1492fff58426 100644

Documentation/userspace-api/media/v4l/subdev-formats.rst The following table lists the existing metadata formats. · line 8241

Quoted 8 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +5 -0
@@ -8234,3 +8234,260 @@ The following table lists the existing metadata formats.
 	both sides of the link and the bus format is a fixed
 	metadata format that is not configurable from userspace.
 	Width and height will be set to 0 for this format.
+
+Generic Serial Metadata Formats
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Generic serial metadata formats are used on serial busses where the actual data

s/busses/buses/

Documentation/userspace-api/media/v4l/subdev-formats.rst The following table lists the existing metadata formats. · line 8247

Quoted 6 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +6 -0
+content is more or less device specific but the data is transmitted and received
+by multiple devices that do not process the data in any way, simply writing
+it to system memory for processing in software at the end of the pipeline.
+
+The more specific variant describing the actual data is used on the internal
+source pad of the originating sub-device.

What do you mean by "more specific variant" here ? Please include an example in the documentation.

Quoted 3 lines

I'm not sure I like mentioning internal source pads here, are we guaranteed that metadata will always originate from an internal source pad ?

We'll have to, otherwise the specific format will be visible on the link, and link validator on the sink end won't (and shouldn't) know about it (and that would systematically lead to a failure to validate it).

Quoted 277 lines
Quoted 253 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +246 -0
+
+"b" in an array cell signifies a byte of data, followed by the number of byte
+and finally the bit number in subscript. "p" indicates a padding bit.
+
+.. _media-bus-format-generic-meta:
+
+.. cssclass: longtable
+
+.. flat-table:: Generic Serial Metadata Formats
+    :header-rows:  2
+    :stub-columns: 0
+
+    * - Identifier
+      - Code
+      -
+      - :cspan:`23` Data organization
+    * -
+      -
+      - Bit
+      - 23
+      - 22
+      - 21
+      - 20
+      - 19
+      - 18
+      - 17
+      - 16
+      - 15
+      - 14
+      - 13
+      - 12
+      - 11
+      - 10
+      - 9
+      - 8
+      - 7
+      - 6
+      - 5
+      - 4
+      - 3
+      - 2
+      - 1
+      - 0
+    * .. _MEDIA-BUS-FMT-META-1X8-8:
+
+      - MEDIA_BUS_FMT_META_1X8_8
+      - 0x8001
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+    * .. _MEDIA-BUS-FMT-META-1X8-10:
+
+      - MEDIA_BUS_FMT_META_1X8_10
+      - 0x8002
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-12:
+
+      - MEDIA_BUS_FMT_META_1X8_12
+      - 0x8003
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-14:
+
+      - MEDIA_BUS_FMT_META_1X8_14
+      - 0x8004
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-16:
+
+      - MEDIA_BUS_FMT_META_1X8_16
+      - 0x8005
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-20:
+
+      - MEDIA_BUS_FMT_META_1X8_20
+      - 0x8007
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-24:
+
+      - MEDIA_BUS_FMT_META_1X8_24
+      - 0x8009
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
include/uapi/linux/media-bus-format.h +0 -0
index a03c543cb072..722463523bbd 100644
@@ -173,4 +173,13 @@
  */

include/uapi/linux/media-bus-format.h line 183

Quoted 10 lines
include/uapi/linux/media-bus-format.h +8 -0
 #define MEDIA_BUS_FMT_METADATA_FIXED		0x7001
 
+/* Generic line based metadata formats for serial buses. Next is 0x800b. */
+#define MEDIA_BUS_FMT_META_1X8_8		0x8001
+#define MEDIA_BUS_FMT_META_1X8_10		0x8002
+#define MEDIA_BUS_FMT_META_1X8_12		0x8003
+#define MEDIA_BUS_FMT_META_1X8_14		0x8004
+#define MEDIA_BUS_FMT_META_1X8_16		0x8005
+#define MEDIA_BUS_FMT_META_1X8_20		0x8007
+#define MEDIA_BUS_FMT_META_1X8_24		0x8009

We've discussed this before privately, it's time to come to a conclusion :-)

My preference would be

#define MEDIA_BUS_FMT_META_RAW8 0x8001
#define MEDIA_BUS_FMT_META_RAW10 0x8002
#define MEDIA_BUS_FMT_META_RAW12 0x8003
#define MEDIA_BUS_FMT_META_RAW14 0x8004
#define MEDIA_BUS_FMT_META_RAW16 0x8005
#define MEDIA_BUS_FMT_META_RAW20 0x8007
#define MEDIA_BUS_FMT_META_RAW24 0x8009

Quoted 2 lines

without defining the contents of the data (that is, no mention of padding bits).

I don't object but I'd like to know what others think.

If we ever need formats with padding (or other depths than 8) we'll need to change the naming. This may still be unlikely.

Signature · 3 lines
Regards,

Sakari Ailus
#35 lore raw

Re: [RFC 5/7] media: uapi: Add generic serial metadata mbus formats

In reply to Sakari Ailus
Quoted 23 lines · Sakari Ailus

Add generic serial metadata mbus formats. These formats describe data width and packing but not the content itself. The reason for specifying such formats is that the formats as such are fairly device specific but they are still handled by CSI-2 receiver drivers that should not be aware of device specific formats. What makes generic metadata formats possible is that these formats are parsed by software only, after capturing the data to system memory.

---
 .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++++++
 include/uapi/linux/media-bus-format.h         |   9 +
 2 files changed, 266 insertions(+)
Documentation/userspace-api/media/v4l/subdev-formats.rst +1 -0
index a3a35eeed708..1492fff58426 100644
@@ -8234,3 +8234,260 @@ The following table lists the existing metadata formats.
 	both sides of the link and the bus format is a fixed
 	metadata format that is not configurable from userspace.
 	Width and height will be set to 0 for this format.
+

Documentation/userspace-api/media/v4l/subdev-formats.rst The following table lists the existing metadata formats. · line 8247

Quoted 10 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +10 -0
+Generic Serial Metadata Formats
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Generic serial metadata formats are used on serial busses where the actual data
+content is more or less device specific but the data is transmitted and received
+by multiple devices that do not process the data in any way, simply writing
+it to system memory for processing in software at the end of the pipeline.
+
+The more specific variant describing the actual data is used on the internal
+source pad of the originating sub-device.

As Laurent said as well, the mention of 'internal source pad' is confusing.

Aren't there two possibilities here? Either the entity has just a source pad that delivers the metadata, and that source pad has the precise metadata format. Or the entity supports routing and then there is a sink pad connected to an internal source that is routed to a source pad. In that case that sink pad has the precise metadata format.

Apologies if I am wrong, it's hard to keep track of all the possibilities, especially if you do not use this regularly.

Quoted 251 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +246 -0
+
+"b" in an array cell signifies a byte of data, followed by the number of byte
+and finally the bit number in subscript. "p" indicates a padding bit.
+
+.. _media-bus-format-generic-meta:
+
+.. cssclass: longtable
+
+.. flat-table:: Generic Serial Metadata Formats
+    :header-rows:  2
+    :stub-columns: 0
+
+    * - Identifier
+      - Code
+      -
+      - :cspan:`23` Data organization
+    * -
+      -
+      - Bit
+      - 23
+      - 22
+      - 21
+      - 20
+      - 19
+      - 18
+      - 17
+      - 16
+      - 15
+      - 14
+      - 13
+      - 12
+      - 11
+      - 10
+      - 9
+      - 8
+      - 7
+      - 6
+      - 5
+      - 4
+      - 3
+      - 2
+      - 1
+      - 0
+    * .. _MEDIA-BUS-FMT-META-1X8-8:
+
+      - MEDIA_BUS_FMT_META_1X8_8
+      - 0x8001
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+    * .. _MEDIA-BUS-FMT-META-1X8-10:
+
+      - MEDIA_BUS_FMT_META_1X8_10
+      - 0x8002
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-12:
+
+      - MEDIA_BUS_FMT_META_1X8_12
+      - 0x8003
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-14:
+
+      - MEDIA_BUS_FMT_META_1X8_14
+      - 0x8004
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-16:
+
+      - MEDIA_BUS_FMT_META_1X8_16
+      - 0x8005
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-20:
+
+      - MEDIA_BUS_FMT_META_1X8_20
+      - 0x8007
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-24:
+
+      - MEDIA_BUS_FMT_META_1X8_24
+      - 0x8009
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
include/uapi/linux/media-bus-format.h +0 -0
index a03c543cb072..722463523bbd 100644

include/uapi/linux/media-bus-format.h line 181

Quoted 9 lines
include/uapi/linux/media-bus-format.h +6 -0
@@ -173,4 +173,13 @@
  */
 #define MEDIA_BUS_FMT_METADATA_FIXED		0x7001
 
+/* Generic line based metadata formats for serial buses. Next is 0x800b. */
+#define MEDIA_BUS_FMT_META_1X8_8		0x8001
+#define MEDIA_BUS_FMT_META_1X8_10		0x8002
+#define MEDIA_BUS_FMT_META_1X8_12		0x8003
+#define MEDIA_BUS_FMT_META_1X8_14		0x8004
+#define MEDIA_BUS_FMT_META_1X8_16		0x8005

You skip 0x8006 here in case there will be a _18 variant?

include/uapi/linux/media-bus-format.h line 185

Quoted 4 lines
include/uapi/linux/media-bus-format.h +3 -0
+#define MEDIA_BUS_FMT_META_1X8_20		0x8007
+#define MEDIA_BUS_FMT_META_1X8_24		0x8009
+
 #endif /* __LINUX_MEDIA_BUS_FORMAT_H */

Regards,

Hans

author maintainer
#36 lore raw

Re: [RFC 5/7] media: uapi: Add generic serial metadata mbus formats

Hi Hans,

Quoted 37 lines · Hans Verkuil
Quoted 23 lines · Sakari Ailus

Add generic serial metadata mbus formats. These formats describe data width and packing but not the content itself. The reason for specifying such formats is that the formats as such are fairly device specific but they are still handled by CSI-2 receiver drivers that should not be aware of device specific formats. What makes generic metadata formats possible is that these formats are parsed by software only, after capturing the data to system memory.

---
 .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++++++
 include/uapi/linux/media-bus-format.h         |   9 +
 2 files changed, 266 insertions(+)
Documentation/userspace-api/media/v4l/subdev-formats.rst +1 -0
index a3a35eeed708..1492fff58426 100644
@@ -8234,3 +8234,260 @@ The following table lists the existing metadata formats.
 	both sides of the link and the bus format is a fixed
 	metadata format that is not configurable from userspace.
 	Width and height will be set to 0 for this format.
+

Documentation/userspace-api/media/v4l/subdev-formats.rst The following table lists the existing metadata formats. · line 8247

Quoted 10 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +10 -0
+Generic Serial Metadata Formats
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Generic serial metadata formats are used on serial busses where the actual data
+content is more or less device specific but the data is transmitted and received
+by multiple devices that do not process the data in any way, simply writing
+it to system memory for processing in software at the end of the pipeline.
+
+The more specific variant describing the actual data is used on the internal
+source pad of the originating sub-device.

As Laurent said as well, the mention of 'internal source pad' is confusing.

Quoted 8 lines

Aren't there two possibilities here? Either the entity has just a source pad that delivers the metadata, and that source pad has the precise metadata format. Or the entity supports routing and then there is a sink pad connected to an internal source that is routed to a source pad. In that case that sink pad has the precise metadata format.

Apologies if I am wrong, it's hard to keep track of all the possibilities, especially if you do not use this regularly.

It's required the generic metadata format is present only on internal pads, otherwise link validation will fail --- the next device won't be aware of the specific mbus code.

Quoted 255 lines
Quoted 255 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +246 -0
+
+"b" in an array cell signifies a byte of data, followed by the number of byte
+and finally the bit number in subscript. "p" indicates a padding bit.
+
+.. _media-bus-format-generic-meta:
+
+.. cssclass: longtable
+
+.. flat-table:: Generic Serial Metadata Formats
+    :header-rows:  2
+    :stub-columns: 0
+
+    * - Identifier
+      - Code
+      -
+      - :cspan:`23` Data organization
+    * -
+      -
+      - Bit
+      - 23
+      - 22
+      - 21
+      - 20
+      - 19
+      - 18
+      - 17
+      - 16
+      - 15
+      - 14
+      - 13
+      - 12
+      - 11
+      - 10
+      - 9
+      - 8
+      - 7
+      - 6
+      - 5
+      - 4
+      - 3
+      - 2
+      - 1
+      - 0
+    * .. _MEDIA-BUS-FMT-META-1X8-8:
+
+      - MEDIA_BUS_FMT_META_1X8_8
+      - 0x8001
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+    * .. _MEDIA-BUS-FMT-META-1X8-10:
+
+      - MEDIA_BUS_FMT_META_1X8_10
+      - 0x8002
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-12:
+
+      - MEDIA_BUS_FMT_META_1X8_12
+      - 0x8003
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-14:
+
+      - MEDIA_BUS_FMT_META_1X8_14
+      - 0x8004
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-16:
+
+      - MEDIA_BUS_FMT_META_1X8_16
+      - 0x8005
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-20:
+
+      - MEDIA_BUS_FMT_META_1X8_20
+      - 0x8007
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-24:
+
+      - MEDIA_BUS_FMT_META_1X8_24
+      - 0x8009
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
include/uapi/linux/media-bus-format.h +0 -0
index a03c543cb072..722463523bbd 100644
@@ -173,4 +173,13 @@
  */
 #define MEDIA_BUS_FMT_METADATA_FIXED		0x7001
 
Quoted 8 lines

include/uapi/linux/media-bus-format.h line 181

Quoted 6 lines
include/uapi/linux/media-bus-format.h +6 -0
+/* Generic line based metadata formats for serial buses. Next is 0x800b. */
+#define MEDIA_BUS_FMT_META_1X8_8		0x8001
+#define MEDIA_BUS_FMT_META_1X8_10		0x8002
+#define MEDIA_BUS_FMT_META_1X8_12		0x8003
+#define MEDIA_BUS_FMT_META_1X8_14		0x8004
+#define MEDIA_BUS_FMT_META_1X8_16		0x8005

You skip 0x8006 here in case there will be a _18 variant?

I removed duplicate entries but forgot to change the values. I'll address that for v2.

There are different packings defined for these formats, sensor generally using the pixel data equivalent format. With 16 and more bits per sample, multiple octets of metadata can be transported in that sample, and those formats are equivalent to the formats at half (or third) of the sample size. Say, alternative format for 16 bpp image data to 1X8_16 is 1X8_8.

Signature · 3 lines
Kind regards,

Sakari Ailus
#37 lore raw

Re: [RFC 5/7] media: uapi: Add generic serial metadata mbus formats

In reply to Sakari Ailus
Quoted 23 lines · Sakari Ailus

Add generic serial metadata mbus formats. These formats describe data width and packing but not the content itself. The reason for specifying such formats is that the formats as such are fairly device specific but they are still handled by CSI-2 receiver drivers that should not be aware of device specific formats. What makes generic metadata formats possible is that these formats are parsed by software only, after capturing the data to system memory.

---
 .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++++++
 include/uapi/linux/media-bus-format.h         |   9 +
 2 files changed, 266 insertions(+)
Documentation/userspace-api/media/v4l/subdev-formats.rst +1 -0
index a3a35eeed708..1492fff58426 100644
@@ -8234,3 +8234,260 @@ The following table lists the existing metadata formats.
 	both sides of the link and the bus format is a fixed
 	metadata format that is not configurable from userspace.
 	Width and height will be set to 0 for this format.
+

Documentation/userspace-api/media/v4l/subdev-formats.rst The following table lists the existing metadata formats. · line 8247

Quoted 10 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +10 -0
+Generic Serial Metadata Formats
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Generic serial metadata formats are used on serial busses where the actual data
+content is more or less device specific but the data is transmitted and received
+by multiple devices that do not process the data in any way, simply writing
+it to system memory for processing in software at the end of the pipeline.
+
+The more specific variant describing the actual data is used on the internal
+source pad of the originating sub-device.

I'm a bit confused here: on which pads are these generic mbus formats set? And who sets this?

I.e. the internal source pad has e.g. MEDIA_BUS_FMT_META_FOOBAR_HIST as specific format. It's routed to a source pad + stream pair for transfer over CSI. Does that source pad + stream pair use the generic mediabus format?

Regards,

Hans

Quoted 264 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +246 -0
+
+"b" in an array cell signifies a byte of data, followed by the number of byte
+and finally the bit number in subscript. "p" indicates a padding bit.
+
+.. _media-bus-format-generic-meta:
+
+.. cssclass: longtable
+
+.. flat-table:: Generic Serial Metadata Formats
+    :header-rows:  2
+    :stub-columns: 0
+
+    * - Identifier
+      - Code
+      -
+      - :cspan:`23` Data organization
+    * -
+      -
+      - Bit
+      - 23
+      - 22
+      - 21
+      - 20
+      - 19
+      - 18
+      - 17
+      - 16
+      - 15
+      - 14
+      - 13
+      - 12
+      - 11
+      - 10
+      - 9
+      - 8
+      - 7
+      - 6
+      - 5
+      - 4
+      - 3
+      - 2
+      - 1
+      - 0
+    * .. _MEDIA-BUS-FMT-META-1X8-8:
+
+      - MEDIA_BUS_FMT_META_1X8_8
+      - 0x8001
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+    * .. _MEDIA-BUS-FMT-META-1X8-10:
+
+      - MEDIA_BUS_FMT_META_1X8_10
+      - 0x8002
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-12:
+
+      - MEDIA_BUS_FMT_META_1X8_12
+      - 0x8003
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-14:
+
+      - MEDIA_BUS_FMT_META_1X8_14
+      - 0x8004
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-16:
+
+      - MEDIA_BUS_FMT_META_1X8_16
+      - 0x8005
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-20:
+
+      - MEDIA_BUS_FMT_META_1X8_20
+      - 0x8007
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-1X8-24:
+
+      - MEDIA_BUS_FMT_META_1X8_24
+      - 0x8009
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
include/uapi/linux/media-bus-format.h +9 -0
index a03c543cb072..722463523bbd 100644
@@ -173,4 +173,13 @@
  */
 #define MEDIA_BUS_FMT_METADATA_FIXED		0x7001
 
+/* Generic line based metadata formats for serial buses. Next is 0x800b. */
+#define MEDIA_BUS_FMT_META_1X8_8		0x8001
+#define MEDIA_BUS_FMT_META_1X8_10		0x8002
+#define MEDIA_BUS_FMT_META_1X8_12		0x8003
+#define MEDIA_BUS_FMT_META_1X8_14		0x8004
+#define MEDIA_BUS_FMT_META_1X8_16		0x8005
+#define MEDIA_BUS_FMT_META_1X8_20		0x8007
+#define MEDIA_BUS_FMT_META_1X8_24		0x8009
+
 #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
author maintainer
#38 lore raw

Re: [RFC 5/7] media: uapi: Add generic serial metadata mbus formats

Hi Hans,

Quoted 29 lines · Hans Verkuil
Quoted 28 lines · Sakari Ailus

Add generic serial metadata mbus formats. These formats describe data width and packing but not the content itself. The reason for specifying such formats is that the formats as such are fairly device specific but they are still handled by CSI-2 receiver drivers that should not be aware of device specific formats. What makes generic metadata formats possible is that these formats are parsed by software only, after capturing the data to system memory.

---
 .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++++++
 include/uapi/linux/media-bus-format.h         |   9 +
 2 files changed, 266 insertions(+)
Documentation/userspace-api/media/v4l/subdev-formats.rst +6 -0
index a3a35eeed708..1492fff58426 100644
@@ -8234,3 +8234,260 @@ The following table lists the existing metadata formats.
 	both sides of the link and the bus format is a fixed
 	metadata format that is not configurable from userspace.
 	Width and height will be set to 0 for this format.
+
+Generic Serial Metadata Formats
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Generic serial metadata formats are used on serial busses where the actual data
+content is more or less device specific but the data is transmitted and received
Quoted 8 lines

Documentation/userspace-api/media/v4l/subdev-formats.rst The following table lists the existing metadata formats. · line 8247

Quoted 5 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +5 -0
+by multiple devices that do not process the data in any way, simply writing
+it to system memory for processing in software at the end of the pipeline.
+
+The more specific variant describing the actual data is used on the internal
+source pad of the originating sub-device.

I'm a bit confused here: on which pads are these generic mbus formats set? And who sets this?

As with other pads, the user can set the format. In most cases only a single option is available whenever the device ourputs metadata over a stream (and thus uses on of the generic line based metadata formats).

Quoted 5 lines

I.e. the internal source pad has e.g. MEDIA_BUS_FMT_META_FOOBAR_HIST as specific format. It's routed to a source pad + stream pair for transfer over CSI. Does that source pad + stream pair use the generic mediabus format?

Correct.

Signature · 1 line
Sakari Ailus
author maintainer
#39 lore raw

RFC 6/7 media: uapi: Add generic 8-bit metadata format definitions

In reply to Sakari Ailus

Generic 8-bit metadata formats define the in-memory data layout but not the format of the data itself. The reasoning for having such formats is to allow CSI-2 receiver drivers to receive and DMA drivers to write the data to memory without knowing a large number of device specific formats.

These formats may be used only in conjunction of a Media controller pipeline where the internal pad of the source sub-device defines the specific format of the data (using an mbus code).

---
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 317 ++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
 include/uapi/linux/videodev2.h                |   9 +
 4 files changed, 335 insertions(+)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
Documentation/userspace-api/media/v4l/meta-formats.rst +1 -0
index 0bb61fc5bc00..919f595576b9 100644
@@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
     metafmt-vsp1-hgo
     metafmt-vsp1-hgt
     metafmt-vivid
+    metafmt-generic
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +317 -0

317 lines not on this page · raw message

drivers/media/v4l2-core/v4l2-ioctl.c +8 -0
index a858acea6547..7781b0bd3e7d 100644
@@ -1451,6 +1451,14 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
+	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8b Generic Meta, 10b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8b Generic Meta, 12b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8b Generic Meta, 14b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8b Generic Meta, 16b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8b Generic Meta, 20b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8b Generic Meta, 24b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_2_24:	descr = "2x8b Generic Meta, 24b CSI-2"; break;
 
 	default:
 		/* Compressed formats */
include/uapi/linux/videodev2.h +9 -0
index aee75eb9e686..adcbdc15dcdb 100644
@@ -834,6 +834,15 @@ struct v4l2_pix_format {
 #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
 #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
 
+#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_2_24	v4l2_fourcc('M', 'C', '2', 'O') /* 2 bytes of 8-bit metadata, 24-bit CSI-2 packed */
+
 /* priv field value to indicates that subsequent fields are valid. */
 #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
 
Signature · 1 line
2.30.2
#40 lore raw

Re: [RFC 6/7] media: uapi: Add generic 8-bit metadata format definitions

Quoted 50 lines · Sakari Ailus

Generic 8-bit metadata formats define the in-memory data layout but not the format of the data itself. The reasoning for having such formats is to allow CSI-2 receiver drivers to receive and DMA drivers to write the data to memory without knowing a large number of device specific formats.

These formats may be used only in conjunction of a Media controller pipeline where the internal pad of the source sub-device defines the specific format of the data (using an mbus code).

---
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 317 ++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
 include/uapi/linux/videodev2.h                |   9 +
 4 files changed, 335 insertions(+)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
Documentation/userspace-api/media/v4l/meta-formats.rst +1 -0
index 0bb61fc5bc00..919f595576b9 100644
@@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
     metafmt-vsp1-hgo
     metafmt-vsp1-hgt
     metafmt-vivid
+    metafmt-generic
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +17 -0
new file mode 100644
index 000000000000..09d3fabb1e94
@@ -0,0 +1,317 @@
+.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
+
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O')
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+
+
+Generic line-based metadata formats
+
+
+Description
+===========
+
+These generic line-based metadata formats define the memory layout of the data
+without defining the format or meaning of the metadata itself. These formats may
+only be used with a Media controller pipeline where the more specific format is
+defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL-SOURCE>` of the

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 27

Quoted 10 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +10 -0
+source sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`.
+
+.. _v4l2-meta-fmt-generic-8:
+
+V4L2_META_FMT_GENERIC_8
+-----------------------
+
+The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format whereas the
+rest of the formats have CSI-2 packing that are typically used on camera sensors
+in conjunction with pixel data of the same bit depth.

You keep referring to CSI-2 packing without ever defining it.

I'm not sure you should include CSI2 in the name either. It is OK to say something here like: "The way the data is stored follows the CSI-2 specification."

But it is not limited to CSI-2 and can be used with other buses as well.

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 30

Quoted 3 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +3 -0
+
+This format is also used on CSI-2 on both 8 bits per sample as well as on
+16 bits per sample when two bytes of metadata are packed into one sample.

Little or big endian?

Regards,

Hans

Quoted 326 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +287 -0
+
+**Byte Order Of V4L2_META_FMT_GENERIC_8.**
+Each cell is one byte. "M" denotes a byte of metadata.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+    * - start + 4:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+
+.. _v4l2-meta-fmt-generic-csi2-10:
+
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains 8-bit generic metadata with CSI-2
+packing, 10 bits for each 8 bits of data. Every four bytes of metadata is
+followed by a single byte of padding.
+
+This format is also used on CSI2 on 20 bits per sample format that packs two
+bytes of metadata into one sample.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+    * - start + 5:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-12:
+
+V4L2_META_FMT_GENERIC_CSI2_12
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_12 contains 8-bit generic metadata with CSI-2
+packing, 12 bits for each 8 bits of data. Every four bytes of metadata is
+followed by two bytes of padding.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 6:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-14:
+
+V4L2_META_FMT_GENERIC_CSI2_14
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_14 contains 8-bit generic metadata with CSI-2
+packing, 14 bits for each 8 bits of data. Every four bytes of metadata is
+followed by three bytes of padding.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+      - p
+      - p
+    * - start + 7:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-16:
+
+V4L2_META_FMT_GENERIC_CSI2_16
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_16 contains 8-bit generic metadata with CSI-2
+packing, 16 bits for each 8 bits of data. Every byte of metadata is
+followed by one byte of padding.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - M\ :sub:`10`
+      - p
+      - M\ :sub:`20`
+      - p
+      - M\ :sub:`30`
+      - p
+    * - start + 8:
+      - M\ :sub:`01`
+      - p
+      - M\ :sub:`11`
+      - p
+      - M\ :sub:`21`
+      - p
+      - M\ :sub:`31`
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-20:
+
+V4L2_META_FMT_GENERIC_CSI2_20
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_20 contains 8-bit generic metadata with CSI-2
+packing, 20 bits for each 8 bits of data. Every byte of metadata is followed by
+alternating one and two bytes of padding.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - M\ :sub:`10`
+      - p
+      - p
+      - M\ :sub:`20`
+      - p
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 10:
+      - M\ :sub:`01`
+      - p
+      - M\ :sub:`11`
+      - p
+      - p
+      - M\ :sub:`21`
+      - p
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-24:
+
+V4L2_META_FMT_GENERIC_CSI2_24
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_24 contains 8-bit generic metadata with CSI-2
+packing, 24 bits for each 8 bits of data. Every byte of metadata is followed by
+alternating one and two bytes of padding.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - p
+      - M\ :sub:`10`
+      - p
+      - p
+      - M\ :sub:`20`
+      - p
+      - p
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 12:
+      - M\ :sub:`01`
+      - p
+      - p
+      - M\ :sub:`11`
+      - p
+      - p
+      - M\ :sub:`21`
+      - p
+      - p
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-2-24:
+
+V4L2_META_FMT_GENERIC_CSI2_2_24
+-------------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_2_24 contains 8-bit generic metadata with CSI-2
+packing, 24 bits for each two times 8 bits of data. Every two bytes of metadata
+are followed by one byte of padding.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_2_24.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - p
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+    * - start + 6:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - p
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+
drivers/media/v4l2-core/v4l2-ioctl.c +8 -0
index a858acea6547..7781b0bd3e7d 100644
@@ -1451,6 +1451,14 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
+	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8b Generic Meta, 10b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8b Generic Meta, 12b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8b Generic Meta, 14b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8b Generic Meta, 16b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8b Generic Meta, 20b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8b Generic Meta, 24b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_2_24:	descr = "2x8b Generic Meta, 24b CSI-2"; break;
 
 	default:
 		/* Compressed formats */
include/uapi/linux/videodev2.h +9 -0
index aee75eb9e686..adcbdc15dcdb 100644
@@ -834,6 +834,15 @@ struct v4l2_pix_format {
 #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
 #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
 
+#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_2_24	v4l2_fourcc('M', 'C', '2', 'O') /* 2 bytes of 8-bit metadata, 24-bit CSI-2 packed */
+
 /* priv field value to indicates that subsequent fields are valid. */
 #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
 
author maintainer
#41 lore raw

Re: [RFC 6/7] media: uapi: Add generic 8-bit metadata format definitions

Hi Laurent,

Quoted 64 lines · Hans Verkuil
Quoted 50 lines · Sakari Ailus

Generic 8-bit metadata formats define the in-memory data layout but not the format of the data itself. The reasoning for having such formats is to allow CSI-2 receiver drivers to receive and DMA drivers to write the data to memory without knowing a large number of device specific formats.

These formats may be used only in conjunction of a Media controller pipeline where the internal pad of the source sub-device defines the specific format of the data (using an mbus code).

---
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 317 ++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
 include/uapi/linux/videodev2.h                |   9 +
 4 files changed, 335 insertions(+)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
Documentation/userspace-api/media/v4l/meta-formats.rst +1 -0
index 0bb61fc5bc00..919f595576b9 100644
@@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
     metafmt-vsp1-hgo
     metafmt-vsp1-hgt
     metafmt-vivid
+    metafmt-generic
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +17 -0
new file mode 100644
index 000000000000..09d3fabb1e94
@@ -0,0 +1,317 @@
+.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
+
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O')
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+
+
+Generic line-based metadata formats
+
+
+Description
+===========
+
+These generic line-based metadata formats define the memory layout of the data
+without defining the format or meaning of the metadata itself. These formats may
+only be used with a Media controller pipeline where the more specific format is
+defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL-SOURCE>` of the

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 27

Quoted 10 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +10 -0
+source sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`.
+
+.. _v4l2-meta-fmt-generic-8:
+
+V4L2_META_FMT_GENERIC_8
+-----------------------
+
+The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format whereas the
+rest of the formats have CSI-2 packing that are typically used on camera sensors
+in conjunction with pixel data of the same bit depth.

You keep referring to CSI-2 packing without ever defining it.

Quoted 4 lines

I'm not sure you should include CSI2 in the name either. It is OK to say something here like: "The way the data is stored follows the CSI-2 specification."

But it is not limited to CSI-2 and can be used with other buses as well.

That's true, certainly, but where this packing is most commonly used is CSI-2 (or derives from CSI-2).

Quoted 6 lines

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 30

Quoted 4 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +3 -0
+
+This format is also used on CSI-2 on both 8 bits per sample as well as on
+16 bits per sample when two bytes of metadata are packed into one sample.

Little or big endian?

Little endian. I'll address these for v2.

Signature · 3 lines
Kind regards,

Sakari Ailus
author maintainer
#42 lore raw

RFC 7/7 media: v4l: Support line-based metadata capture

In reply to Sakari Ailus

many camera sensors, among other devices, transmit embedded data and image data for each CSI-2 frame. This embedded data typically contains register configuration of the sensor that has been used to capture the image data of the same frame.

The embedded data is received by the CSI-2 receiver and has the same properties as the image data, including that it is line based: it has width, height and bytesperline (stride).

Add these fields to struct v4l2_meta_format and document them.

Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is line-based i.e. these fields of struct v4l2_meta_format are valid for it.

---
 .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
 .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
 include/uapi/linux/videodev2.h                    | 10 ++++++++++
 3 files changed, 32 insertions(+)
Documentation/userspace-api/media/v4l/dev-meta.rst +15 -0
index 0e7e1ee1471a..7d3a64514db0 100644
@@ -65,3 +65,18 @@ to 0.
       - ``buffersize``
       - Maximum buffer size in bytes required for data. The value is set by the
         driver.
+    * - __u32
+      - ``width``
+      - Width of a line of metadata in bytes. Valid when :c:type`v4l2_fmtdesc`
+	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``height``
+      - Height of a line of metadata in bytes. Valid when :c:type`v4l2_fmtdesc`
+	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``bytesperlines``
+      - Offset in bytes between the beginning of two consecutive lines. Valid
+	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
+	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +7 -0
index 000c154b0f98..6d7664345a4e 100644
@@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
 	The application can ask to configure the quantization of the capture
 	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
 	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
+    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
+      - 0x0200
+      - The metadata format is line-based. In this case the ``width``,
+	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
+	valid. The buffer consists of ``height`` lines, each having ``width``
+	bytes of data and offset between the beginning of each two consecutive
+	lines is ``bytesperline``.
 
 Return Value
 ============
include/uapi/linux/videodev2.h +10 -0
index adcbdc15dcdb..3681b2c15901 100644
@@ -873,6 +873,7 @@ struct v4l2_fmtdesc {
 #define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0080
 #define V4L2_FMT_FLAG_CSC_HSV_ENC		V4L2_FMT_FLAG_CSC_YCBCR_ENC
 #define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0100
+#define V4L2_FMT_FLAG_META_LINE_BASED		0x0200
 
 	/* Frame Size and frame rate enumeration */
 /*
@@ -2407,10 +2408,19 @@ struct v4l2_sdr_format {
  * struct v4l2_meta_format - metadata format definition
  * @dataformat:		little endian four character code (fourcc)
  * @buffersize:		maximum size in bytes required for data
+ * @width:		number of bytes of data per line (valid for line based
+ *			formats only, see format documentation)
+ * @height:		number of lines of data per buffer (valid for line based
+ *			formats only)
+ * @bytesperline:	offset between the beginnings of two adjacent lines in
+ *			bytes (valid for line based formats only)
  */
 struct v4l2_meta_format {
 	__u32				dataformat;
 	__u32				buffersize;
+	__u32				width;
+	__u32				height;
+	__u32				bytesperline;
 } __attribute__ ((packed));
 
 /**
Signature · 1 line
2.30.2
maintainer
#43 lore raw

Re: [RFC 7/7] media: v4l: Support line-based metadata capture

Hi Sakari,

Thank you for the patch.

Quoted 1 line · Sakari Ailus

many camera sensors, among other devices, transmit embedded data and image

s/many/Many/

Quoted 25 lines

data for each CSI-2 frame. This embedded data typically contains register configuration of the sensor that has been used to capture the image data of the same frame.

The embedded data is received by the CSI-2 receiver and has the same properties as the image data, including that it is line based: it has width, height and bytesperline (stride).

Add these fields to struct v4l2_meta_format and document them.

Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is line-based i.e. these fields of struct v4l2_meta_format are valid for it.

---
 .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
 .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
 include/uapi/linux/videodev2.h                    | 10 ++++++++++
 3 files changed, 32 insertions(+)
Documentation/userspace-api/media/v4l/dev-meta.rst +0 -0
index 0e7e1ee1471a..7d3a64514db0 100644

Documentation/userspace-api/media/v4l/dev-meta.rst to 0. · line 70

Quoted 6 lines
Documentation/userspace-api/media/v4l/dev-meta.rst +3 -0
@@ -65,3 +65,18 @@ to 0.
       - ``buffersize``
       - Maximum buffer size in bytes required for data. The value is set by the
         driver.
+    * - __u32
+      - ``width``
+      - Width of a line of metadata in bytes. Valid when :c:type`v4l2_fmtdesc`

This departs from pixel formats, where the width is defined in pixels. I wonder what the implications will be for userspace. Seeing one implementation, both in a kernel driver and in libcamera, will help validating the API.

Documentation/userspace-api/media/v4l/dev-meta.rst to 0. · line 75

Quoted 5 lines
Documentation/userspace-api/media/v4l/dev-meta.rst +5 -0
+	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``height``
+      - Height of a line of metadata in bytes. Valid when :c:type`v4l2_fmtdesc`

The "height of a line" seems like a weird concept, especially if the height is expressed in bytes. I assume this is a bad copy&paste.

Quoted 12 lines
Documentation/userspace-api/media/v4l/dev-meta.rst +7 -0
+	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``bytesperlines``
+      - Offset in bytes between the beginning of two consecutive lines. Valid
+	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
+	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +0 -0
index 000c154b0f98..6d7664345a4e 100644

Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst the ``mbus_code`` field is handled differently: · line 236

Quoted 10 lines
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +7 -0
@@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
 	The application can ask to configure the quantization of the capture
 	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
 	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
+    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
+      - 0x0200
+      - The metadata format is line-based. In this case the ``width``,
+	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
+	valid. The buffer consists of ``height`` lines, each having ``width``
+	bytes of data and offset between the beginning of each two consecutive
+	lines is ``bytesperline``.

If we add width and height for metadata formats, does it mean that drivers have to (or can) implement VIDIOC_ENUM_FRAMESIZES ? This should be documented.

Quoted 35 lines
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +0 -0
 Return Value
 ============
include/uapi/linux/videodev2.h +10 -0
index adcbdc15dcdb..3681b2c15901 100644
@@ -873,6 +873,7 @@ struct v4l2_fmtdesc {
 #define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0080
 #define V4L2_FMT_FLAG_CSC_HSV_ENC		V4L2_FMT_FLAG_CSC_YCBCR_ENC
 #define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0100
+#define V4L2_FMT_FLAG_META_LINE_BASED		0x0200
 
 	/* Frame Size and frame rate enumeration */
 /*
@@ -2407,10 +2408,19 @@ struct v4l2_sdr_format {
  * struct v4l2_meta_format - metadata format definition
  * @dataformat:		little endian four character code (fourcc)
  * @buffersize:		maximum size in bytes required for data
+ * @width:		number of bytes of data per line (valid for line based
+ *			formats only, see format documentation)
+ * @height:		number of lines of data per buffer (valid for line based
+ *			formats only)
+ * @bytesperline:	offset between the beginnings of two adjacent lines in
+ *			bytes (valid for line based formats only)
  */
 struct v4l2_meta_format {
 	__u32				dataformat;
 	__u32				buffersize;
+	__u32				width;
+	__u32				height;
+	__u32				bytesperline;
 } __attribute__ ((packed));
 
 /**
Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#44 lore raw

Re: [RFC 7/7] media: v4l: Support line-based metadata capture

Hi Hans,

Quoted 29 lines · Laurent Pinchart

Hi Sakari,

Thank you for the patch.

Quoted 1 line · Sakari Ailus

many camera sensors, among other devices, transmit embedded data and image

s/many/Many/

Quoted 20 lines

data for each CSI-2 frame. This embedded data typically contains register configuration of the sensor that has been used to capture the image data of the same frame.

The embedded data is received by the CSI-2 receiver and has the same properties as the image data, including that it is line based: it has width, height and bytesperline (stride).

Add these fields to struct v4l2_meta_format and document them.

Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is line-based i.e. these fields of struct v4l2_meta_format are valid for it.

---
 .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
 .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
 include/uapi/linux/videodev2.h                    | 10 ++++++++++
 3 files changed, 32 insertions(+)
Quoted 16 lines

Documentation/userspace-api/media/v4l/dev-meta.rst to 0. · line 70

Quoted 11 lines
Documentation/userspace-api/media/v4l/dev-meta.rst +3 -0
index 0e7e1ee1471a..7d3a64514db0 100644
@@ -65,3 +65,18 @@ to 0.
       - ``buffersize``
       - Maximum buffer size in bytes required for data. The value is set by the
         driver.
+    * - __u32
+      - ``width``
+      - Width of a line of metadata in bytes. Valid when :c:type`v4l2_fmtdesc`

This departs from pixel formats, where the width is defined in pixels. I wonder what the implications will be for userspace. Seeing one implementation, both in a kernel driver and in libcamera, will help validating the API.

I thought of bytes when writing this but pixels (or samples) are probably a better term for this.

Quoted 9 lines

Documentation/userspace-api/media/v4l/dev-meta.rst to 0. · line 75

Quoted 6 lines
Documentation/userspace-api/media/v4l/dev-meta.rst +5 -0
+	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``height``
+      - Height of a line of metadata in bytes. Valid when :c:type`v4l2_fmtdesc`

The "height of a line" seems like a weird concept, especially if the height is expressed in bytes. I assume this is a bad copy&paste.

Yes. I'll address these for v2.

Quoted 19 lines
Quoted 19 lines
Documentation/userspace-api/media/v4l/dev-meta.rst +7 -0
+	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``bytesperlines``
+      - Offset in bytes between the beginning of two consecutive lines. Valid
+	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
+	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +3 -0
index 000c154b0f98..6d7664345a4e 100644
@@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
 	The application can ask to configure the quantization of the capture
 	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
 	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
+    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
+      - 0x0200
+      - The metadata format is line-based. In this case the ``width``,
Quoted 8 lines

Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst the ``mbus_code`` field is handled differently: · line 236

Quoted 4 lines
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +4 -0
+	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
+	valid. The buffer consists of ``height`` lines, each having ``width``
+	bytes of data and offset between the beginning of each two consecutive
+	lines is ``bytesperline``.

If we add width and height for metadata formats, does it mean that drivers have to (or can) implement VIDIOC_ENUM_FRAMESIZES ? This should be documented.

Good point.

I don't think it's meaningful to implement that as the pipeline configuration determines this in any case.

But I wonder whether this would better be documented for V4L2_CAP_IO_MC rather than for metadata formats. There's currently only a reference to this in ENUM_FMT documentation.

Quoted 36 lines
Quoted 36 lines
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +0 -0
 Return Value
 ============
include/uapi/linux/videodev2.h +10 -0
index adcbdc15dcdb..3681b2c15901 100644
@@ -873,6 +873,7 @@ struct v4l2_fmtdesc {
 #define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0080
 #define V4L2_FMT_FLAG_CSC_HSV_ENC		V4L2_FMT_FLAG_CSC_YCBCR_ENC
 #define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0100
+#define V4L2_FMT_FLAG_META_LINE_BASED		0x0200
 
 	/* Frame Size and frame rate enumeration */
 /*
@@ -2407,10 +2408,19 @@ struct v4l2_sdr_format {
  * struct v4l2_meta_format - metadata format definition
  * @dataformat:		little endian four character code (fourcc)
  * @buffersize:		maximum size in bytes required for data
+ * @width:		number of bytes of data per line (valid for line based
+ *			formats only, see format documentation)
+ * @height:		number of lines of data per buffer (valid for line based
+ *			formats only)
+ * @bytesperline:	offset between the beginnings of two adjacent lines in
+ *			bytes (valid for line based formats only)
  */
 struct v4l2_meta_format {
 	__u32				dataformat;
 	__u32				buffersize;
+	__u32				width;
+	__u32				height;
+	__u32				bytesperline;
 } __attribute__ ((packed));
 
 /**
Signature · 3 lines
Regards,

Sakari Ailus
#45 lore raw
In reply to Sakari Ailus

Hi Sakari,

Thank you for working on this. Sensor metadata is something that Raspberry Pi do make extensive use of, and our downstream changes to support it, although a bit hacky, are not too dissimilar to your proposal here.

Quoted 49 lines · Sakari Ailus

Hi folks,

Here are a few patches to add support generic, line based metadata as well as internal source pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.

Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.

Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc. The number of these formats is probably going to be about as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.

Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.

Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).

The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).

Quoted 4 lines

These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.

One thing that may be worth considering or clarifying - for the case of the BCM2835 Unicam CSI-2 device, we only have 2x DMA output channels. So one will match image data packets, and the other will match "everything else". Typically "everything else" would only be CSI-2 embedded data, but in the case of the Raspberry Pi Camera v3 (IMX708), it includes embedded data, PDAF data, and HDR histogram data. Each of these outputs can be programmed to use a different packet ID in the sensor, but since Unicam only has a single DMA for "everything else", it all gets dumped into one metadata buffer. But given we know the exact structure of the data streams, it's trivial for useland to find the right bits in this buffer. Of course, other CSI-2 receivers with more DMA channels might allow these streams to end up in their own buffers.

Nothing in your series seems to stop us operating Unicam in this way, particularly because there is no fixed definition of the data format for V4L2_META_FMT_GENERIC_8. So I don't think it's a problem, but perhaps it's worth documenting that the metadata might include multiple streams from the sensor?

Regards,
Naush

Quoted 57 lines

The internal (source) pads are an alternative to source routes [1]. The source routes were not universally liked and I do have to say I like re-using existing interface concepts (pads and everything you can do with pads, including access format, selections etc.) wherever it makes sense, instead of duplicating functionality.

Effectively internal source pads behave mostly just like sink pads, but they describe a flow of data that originates from a sub-device instead of arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable and disable routes from internal source pads to sub-device's source pads. The subdev format IOCTLs are usable, too, so one can find which subdev format is available on given internal source pad.

This set depends on these patches:

<URL:https://lore.kernel.org/linux-media/[email protected]/T/#t>

I've also pushed these here and I'll keep updating the branch:

<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>

Questions and comments are most welcome.

[1] <URL:https://lore.kernel.org/linux-media/[email protected]/>

Sakari Ailus (7):
media: mc: Add INTERNAL_SOURCE pad type flag
media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs
media: uapi: v4l: Document source routes
media: mc: Check pad flag validity
media: uapi: Add generic serial metadata mbus formats
media: uapi: Add generic 8-bit metadata format definitions
media: v4l: Support line-based metadata capture

 .../media/mediactl/media-types.rst            |   7 +
 .../userspace-api/media/v4l/dev-meta.rst      |  15 +
 .../userspace-api/media/v4l/dev-subdev.rst    |  18 +
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 317 ++++++++++++++++++
 .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++
 .../media/v4l/vidioc-enum-fmt.rst             |   7 +
 .../media/v4l/vidioc-subdev-g-routing.rst     |   5 +
 drivers/media/mc/mc-entity.c                  |  20 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
 drivers/media/v4l2-core/v4l2-subdev.c         |   6 +-
 include/uapi/linux/media-bus-format.h         |   9 +
 include/uapi/linux/media.h                    |   1 +
 include/uapi/linux/v4l2-subdev.h              |   6 +-
 include/uapi/linux/videodev2.h                |  19 ++
 15 files changed, 691 insertions(+), 5 deletions(-)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst

--
2.30.2

author maintainer
#46 lore raw

Hi Naushir,

Quoted 75 lines · Naushir Patuck

Hi Sakari,

Thank you for working on this. Sensor metadata is something that Raspberry Pi do make extensive use of, and our downstream changes to support it, although a bit hacky, are not too dissimilar to your proposal here.

Quoted 49 lines · Sakari Ailus

Hi folks,

Here are a few patches to add support generic, line based metadata as well as internal source pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.

Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.

Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc. The number of these formats is probably going to be about as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.

Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.

Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).

The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).

Quoted 4 lines

These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.

One thing that may be worth considering or clarifying - for the case of the BCM2835 Unicam CSI-2 device, we only have 2x DMA output channels. So one will match image data packets, and the other will match "everything else". Typically "everything else" would only be CSI-2 embedded data, but in the case of the Raspberry Pi Camera v3 (IMX708), it includes embedded data, PDAF data, and HDR histogram data. Each of these outputs can be programmed to use a different packet ID in the sensor, but since Unicam only has a single DMA for "everything else", it all gets dumped into one metadata buffer. But given we know the exact structure of the data streams, it's trivial for useland to find the right bits in this buffer. Of course, other CSI-2 receivers with more DMA channels might allow these streams to end up in their own buffers.

Quoted 5 lines

Nothing in your series seems to stop us operating Unicam in this way, particularly because there is no fixed definition of the data format for V4L2_META_FMT_GENERIC_8. So I don't think it's a problem, but perhaps it's worth documenting that the metadata might include multiple streams from the sensor?

I believe this happens on other hardware, too, indeed. Currently the documentation says that

Any number of routes from streams on sink pads towards streams on source pads is allowed, to the extent supported by drivers. For every stream on a source pad, however, only a single route is allowed.

(Documentation/userspace-api/media/v4l/dev-subdev.rst)

This probably needs to be changed to allow what you'd need?

Signature · 3 lines
Kind regards,

Sakari Ailus
#47 lore raw

Hi Sakari,

Quoted 88 lines · Sakari Ailus

Hi Naushir,

Quoted 75 lines · Naushir Patuck

Hi Sakari,

Thank you for working on this. Sensor metadata is something that Raspberry Pi do make extensive use of, and our downstream changes to support it, although a bit hacky, are not too dissimilar to your proposal here.

Quoted 49 lines · Sakari Ailus

Hi folks,

Here are a few patches to add support generic, line based metadata as well as internal source pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.

Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.

Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc. The number of these formats is probably going to be about as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.

Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.

Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).

The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).

Quoted 4 lines

These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.

One thing that may be worth considering or clarifying - for the case of the BCM2835 Unicam CSI-2 device, we only have 2x DMA output channels. So one will match image data packets, and the other will match "everything else". Typically "everything else" would only be CSI-2 embedded data, but in the case of the Raspberry Pi Camera v3 (IMX708), it includes embedded data, PDAF data, and HDR histogram data. Each of these outputs can be programmed to use a different packet ID in the sensor, but since Unicam only has a single DMA for "everything else", it all gets dumped into one metadata buffer. But given we know the exact structure of the data streams, it's trivial for useland to find the right bits in this buffer. Of course, other CSI-2 receivers with more DMA channels might allow these streams to end up in their own buffers.

Quoted 5 lines

Nothing in your series seems to stop us operating Unicam in this way, particularly because there is no fixed definition of the data format for V4L2_META_FMT_GENERIC_8. So I don't think it's a problem, but perhaps it's worth documenting that the metadata might include multiple streams from the sensor?

I believe this happens on other hardware, too, indeed. Currently the documentation says that

Quoted 8 lines

Any number of routes from streams on sink pads towards streams on source pads is allowed, to the extent supported by drivers. For every stream on a source pad, however, only a single route is allowed.

(Documentation/userspace-api/media/v4l/dev-subdev.rst)

This probably needs to be changed to allow what you'd need?

Yes, that last sentence sounds like it would (artificially wrt your series) limit
metadata buffers to only handle a single output stream. However, I may have got the context of the paragraph wrong as well :)

Regards,
Naush

Quoted 5 lines

--
Kind regards,

Sakari Ailus

author maintainer
#48 lore raw

Hi Naush,

Quoted 96 lines · Naushir Patuck

Hi Sakari,

Quoted 93 lines · Sakari Ailus

Hi Naushir,

Quoted 75 lines · Naushir Patuck

Hi Sakari,

Thank you for working on this. Sensor metadata is something that Raspberry Pi do make extensive use of, and our downstream changes to support it, although a bit hacky, are not too dissimilar to your proposal here.

Quoted 49 lines · Sakari Ailus

Hi folks,

Here are a few patches to add support generic, line based metadata as well as internal source pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.

Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.

Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc. The number of these formats is probably going to be about as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.

Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.

Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).

The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).

Quoted 4 lines

These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.

One thing that may be worth considering or clarifying - for the case of the BCM2835 Unicam CSI-2 device, we only have 2x DMA output channels. So one will match image data packets, and the other will match "everything else". Typically "everything else" would only be CSI-2 embedded data, but in the case of the Raspberry Pi Camera v3 (IMX708), it includes embedded data, PDAF data, and HDR histogram data. Each of these outputs can be programmed to use a different packet ID in the sensor, but since Unicam only has a single DMA for "everything else", it all gets dumped into one metadata buffer. But given we know the exact structure of the data streams, it's trivial for useland to find the right bits in this buffer. Of course, other CSI-2 receivers with more DMA channels might allow these streams to end up in their own buffers.

Quoted 5 lines

Nothing in your series seems to stop us operating Unicam in this way, particularly because there is no fixed definition of the data format for V4L2_META_FMT_GENERIC_8. So I don't think it's a problem, but perhaps it's worth documenting that the metadata might include multiple streams from the sensor?

I believe this happens on other hardware, too, indeed. Currently the documentation says that

Any number of routes from streams on sink pads towards streams on source pads is allowed, to the extent supported by drivers. For every stream on a source pad, however, only a single route is allowed.

Quoted 7 lines
Quoted 3 lines

(Documentation/userspace-api/media/v4l/dev-subdev.rst)

This probably needs to be changed to allow what you'd need?

Yes, that last sentence sounds like it would (artificially wrt your series) limit metadata buffers to only handle a single output stream. However, I may have got the context of the paragraph wrong as well :)

That was exactly the purpose: I wanted to make sure we didn't allow this without thinking what other implications it could have --- for instance what you also mentioned, the V4L2 format of the related buffer.

Signature · 3 lines
Kind regards,

Sakari Ailus
maintainer
#49 lore raw
In reply to Naushir Patuck

Hello,

Quoted 25 lines · Naushir Patuck

Hi Sakari,

Thank you for working on this. Sensor metadata is something that Raspberry Pi do make extensive use of, and our downstream changes to support it, although a bit hacky, are not too dissimilar to your proposal here.

Quoted 18 lines · Sakari Ailus

Hi folks,

Here are a few patches to add support generic, line based metadata as well as internal source pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.

Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.

Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc.

Optical dark pixels are image data, I wouldn't include them in the "metadata" category. They can of course be transmitted over a different stream, so they're relevant to the API being designed.

Quoted 5 lines
Quoted 5 lines

The number of these formats is probably going to be about
as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.

Strictly speaking, the number of metadata formats depends on how we define "format". Today, we can use the GREY pixel format to capture greyscale images in the visible spectrum, but also IR images, thermal images, or even depth images. They're all one pixel format. On the other hand, we have Y16 for greyscale visible and IR images, and Z16 for depth maps. It's already a mess, even without metadata :-)

Quoted 10 lines
Quoted 10 lines

Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.

I'm glad we agree on this.

Quoted 7 lines
Quoted 7 lines

Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).

Decoupling the information needed by the kernel (e.g. are we transporting RAW8 or RAW10 from the sensor through the pipeline) from the information useful for userspace only (e.g. the sensor embedded data is encoding using register + value pairs, based on the IMX708 registers set) is a good idea. I expect the main question to be where to draw the line between those two categories. Some pieces of information may be useless to any processing block in the pipeline except for an odd block in the middle.

This is, I believe, a problem similar to the CFA pattern. That information is useless for most devices, but required by the demosaicing block and some other blocks along the pipeline (colour gains for instance, or some Bayer statistics engines). We currently convey the CFA pattern in the media bus codes and pixel formats (e.g. SGRBG8 vs. SRGGB8) through the whole pipeline, while it could be conveyed out of band (e.g. exposed by the sensor using a control, and set on the devices that need it using a control as well).

If we come up with a good solution for metadata (and I hope we will), maybe we'll be able to use a similar mechanism for CFA patterns, simplifying new drivers and userspace. Or maybe this will remain a pipe dream given the backward compatibility implications.

Quoted 24 lines
Quoted 11 lines

The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).

These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.

One thing that may be worth considering or clarifying - for the case of the BCM2835 Unicam CSI-2 device, we only have 2x DMA output channels. So one will match image data packets, and the other will match "everything else". Typically "everything else" would only be CSI-2 embedded data, but in the case of the Raspberry Pi Camera v3 (IMX708), it includes embedded data, PDAF data, and HDR histogram data. Each of these outputs can be programmed to use a different packet ID in the sensor, but since Unicam only has a single DMA for "everything else", it all gets dumped into one metadata buffer. But given we know the exact structure of the data streams, it's trivial for useland to find the right bits in this buffer. Of course, other CSI-2 receivers with more DMA channels might allow these streams to end up in their own buffers.

Quoted 4 lines

Nothing in your series seems to stop us operating Unicam in this way, particularly because there is no fixed definition of the data format for V4L2_META_FMT_GENERIC_8. So I don't think it's a problem, but perhaps it's worth documenting that the metadata might include multiple streams from the sensor?

Thanks for your feedback Naush. Would you consider reviewing the individual patches in the series ? :-)

Quoted 52 lines
Quoted 52 lines

The internal (source) pads are an alternative to source routes [1]. The source routes were not universally liked and I do have to say I like re-using existing interface concepts (pads and everything you can do with pads, including access format, selections etc.) wherever it makes sense, instead of duplicating functionality.

Effectively internal source pads behave mostly just like sink pads, but they describe a flow of data that originates from a sub-device instead of arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable and disable routes from internal source pads to sub-device's source pads. The subdev format IOCTLs are usable, too, so one can find which subdev format is available on given internal source pad.

This set depends on these patches:

<URL:https://lore.kernel.org/linux-media/[email protected]/T/#t>

I've also pushed these here and I'll keep updating the branch:

<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>

Questions and comments are most welcome.

[1] <URL:https://lore.kernel.org/linux-media/[email protected]/>

Sakari Ailus (7):
media: mc: Add INTERNAL_SOURCE pad type flag
media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs
media: uapi: v4l: Document source routes
media: mc: Check pad flag validity
media: uapi: Add generic serial metadata mbus formats
media: uapi: Add generic 8-bit metadata format definitions
media: v4l: Support line-based metadata capture

 .../media/mediactl/media-types.rst            |   7 +
 .../userspace-api/media/v4l/dev-meta.rst      |  15 +
 .../userspace-api/media/v4l/dev-subdev.rst    |  18 +
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 317 ++++++++++++++++++
 .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++
 .../media/v4l/vidioc-enum-fmt.rst             |   7 +
 .../media/v4l/vidioc-subdev-g-routing.rst     |   5 +
 drivers/media/mc/mc-entity.c                  |  20 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
 drivers/media/v4l2-core/v4l2-subdev.c         |   6 +-
 include/uapi/linux/media-bus-format.h         |   9 +
 include/uapi/linux/media.h                    |   1 +
 include/uapi/linux/v4l2-subdev.h              |   6 +-
 include/uapi/linux/videodev2.h                |  19 ++
 15 files changed, 691 insertions(+), 5 deletions(-)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
Signature · 3 lines
Regards,

Laurent Pinchart
#50 lore raw
Quoted 115 lines · Laurent Pinchart

Hello,

Quoted 25 lines · Naushir Patuck

Hi Sakari,

Thank you for working on this. Sensor metadata is something that Raspberry Pi do make extensive use of, and our downstream changes to support it, although a bit hacky, are not too dissimilar to your proposal here.

Quoted 18 lines · Sakari Ailus

Hi folks,

Here are a few patches to add support generic, line based metadata as well as internal source pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.

Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.

Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc.

Optical dark pixels are image data, I wouldn't include them in the "metadata" category. They can of course be transmitted over a different stream, so they're relevant to the API being designed.

Quoted 5 lines
Quoted 5 lines

The number of these formats is probably going to be about
as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.

Strictly speaking, the number of metadata formats depends on how we define "format". Today, we can use the GREY pixel format to capture greyscale images in the visible spectrum, but also IR images, thermal images, or even depth images. They're all one pixel format. On the other hand, we have Y16 for greyscale visible and IR images, and Z16 for depth maps. It's already a mess, even without metadata :-)

Quoted 10 lines
Quoted 10 lines

Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.

I'm glad we agree on this.

Quoted 7 lines
Quoted 7 lines

Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).

Decoupling the information needed by the kernel (e.g. are we transporting RAW8 or RAW10 from the sensor through the pipeline) from the information useful for userspace only (e.g. the sensor embedded data is encoding using register + value pairs, based on the IMX708 registers set) is a good idea. I expect the main question to be where to draw the line between those two categories. Some pieces of information may be useless to any processing block in the pipeline except for an odd block in the middle.

This is, I believe, a problem similar to the CFA pattern. That information is useless for most devices, but required by the demosaicing block and some other blocks along the pipeline (colour gains for instance, or some Bayer statistics engines). We currently convey the CFA pattern in the media bus codes and pixel formats (e.g. SGRBG8 vs. SRGGB8) through the whole pipeline, while it could be conveyed out of band (e.g. exposed by the sensor using a control, and set on the devices that need it using a control as well).

If we come up with a good solution for metadata (and I hope we will), maybe we'll be able to use a similar mechanism for CFA patterns, simplifying new drivers and userspace. Or maybe this will remain a pipe dream given the backward compatibility implications.

Quoted 24 lines
Quoted 11 lines

The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).

These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.

One thing that may be worth considering or clarifying - for the case of the BCM2835 Unicam CSI-2 device, we only have 2x DMA output channels. So one will match image data packets, and the other will match "everything else". Typically "everything else" would only be CSI-2 embedded data, but in the case of the Raspberry Pi Camera v3 (IMX708), it includes embedded data, PDAF data, and HDR histogram data. Each of these outputs can be programmed to use a different packet ID in the sensor, but since Unicam only has a single DMA for "everything else", it all gets dumped into one metadata buffer. But given we know the exact structure of the data streams, it's trivial for useland to find the right bits in this buffer. Of course, other CSI-2 receivers with more DMA channels might allow these streams to end up in their own buffers.

Quoted 7 lines
Quoted 4 lines

Nothing in your series seems to stop us operating Unicam in this way, particularly because there is no fixed definition of the data format for V4L2_META_FMT_GENERIC_8. So I don't think it's a problem, but perhaps it's worth documenting that the metadata might include multiple streams from the sensor?

Thanks for your feedback Naush. Would you consider reviewing the individual patches in the series ? :-)

Sure, I'll go through the patches and send some feedback. This is after all a change that I'm very much looking forward to :-)

Regards,
Naush

Quoted 58 lines
Quoted 45 lines
Quoted 45 lines

The internal (source) pads are an alternative to source routes [1]. The source routes were not universally liked and I do have to say I like re-using existing interface concepts (pads and everything you can do with pads, including access format, selections etc.) wherever it makes sense, instead of duplicating functionality.

Effectively internal source pads behave mostly just like sink pads, but they describe a flow of data that originates from a sub-device instead of arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable and disable routes from internal source pads to sub-device's source pads. The subdev format IOCTLs are usable, too, so one can find which subdev format is available on given internal source pad.

This set depends on these patches:

<URL:https://lore.kernel.org/linux-media/[email protected]/T/#t>

I've also pushed these here and I'll keep updating the branch:

<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>

Questions and comments are most welcome.

[1] <URL:https://lore.kernel.org/linux-media/[email protected]/>

Sakari Ailus (7):
media: mc: Add INTERNAL_SOURCE pad type flag
media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs
media: uapi: v4l: Document source routes
media: mc: Check pad flag validity
media: uapi: Add generic serial metadata mbus formats
media: uapi: Add generic 8-bit metadata format definitions
media: v4l: Support line-based metadata capture

 .../media/mediactl/media-types.rst            |   7 +
 .../userspace-api/media/v4l/dev-meta.rst      |  15 +
 .../userspace-api/media/v4l/dev-subdev.rst    |  18 +
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 317 ++++++++++++++++++
 .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++
 .../media/v4l/vidioc-enum-fmt.rst             |   7 +
 .../media/v4l/vidioc-subdev-g-routing.rst     |   5 +
 drivers/media/mc/mc-entity.c                  |  20 +-
Quoted 8 lines
Quoted 8 lines
 drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
 drivers/media/v4l2-core/v4l2-subdev.c         |   6 +-
 include/uapi/linux/media-bus-format.h         |   9 +
 include/uapi/linux/media.h                    |   1 +
 include/uapi/linux/v4l2-subdev.h              |   6 +-
 include/uapi/linux/videodev2.h                |  19 ++
 15 files changed, 691 insertions(+), 5 deletions(-)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst

--
Regards,

Laurent Pinchart

author maintainer
#51 lore raw
In reply to Laurent Pinchart

Hi Laurent,

Quoted 24 lines · Laurent Pinchart

Hello,

Quoted 21 lines · Naushir Patuck

Hi Sakari,

Thank you for working on this. Sensor metadata is something that Raspberry Pi do make extensive use of, and our downstream changes to support it, although a bit hacky, are not too dissimilar to your proposal here.

Quoted 14 lines · Sakari Ailus

Hi folks,

Here are a few patches to add support generic, line based metadata as well as internal source pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.

Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.

Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc.

Optical dark pixels are image data, I wouldn't include them in the "metadata" category. They can of course be transmitted over a different stream, so they're relevant to the API being designed.

Good point, I'll address this for v2.

Quoted 7 lines
Quoted 6 lines
Quoted 6 lines

The number of these formats is probably going to be about
as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.

Quoted 6 lines

Strictly speaking, the number of metadata formats depends on how we define "format". Today, we can use the GREY pixel format to capture greyscale images in the visible spectrum, but also IR images, thermal images, or even depth images. They're all one pixel format. On the other hand, we have Y16 for greyscale visible and IR images, and Z16 for depth maps. It's already a mess, even without metadata :-)

Yes.

Quoted 12 lines
Quoted 11 lines
Quoted 11 lines

Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.

Quoted 1 line

I'm glad we agree on this.

Well, there are alternatives, but this is probably easier in the long run.

We'll just need more documentation than usually for the metadata mbus codes.

Quoted 18 lines
Quoted 8 lines
Quoted 8 lines

Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).

Decoupling the information needed by the kernel (e.g. are we transporting RAW8 or RAW10 from the sensor through the pipeline) from the information useful for userspace only (e.g. the sensor embedded data is encoding using register + value pairs, based on the IMX708 registers set) is a good idea. I expect the main question to be where to draw the line between those two categories. Some pieces of information may be useless to any processing block in the pipeline except for an odd block in the middle.

Quoted 8 lines

This is, I believe, a problem similar to the CFA pattern. That information is useless for most devices, but required by the demosaicing block and some other blocks along the pipeline (colour gains for instance, or some Bayer statistics engines). We currently convey the CFA pattern in the media bus codes and pixel formats (e.g. SGRBG8 vs. SRGGB8) through the whole pipeline, while it could be conveyed out of band (e.g. exposed by the sensor using a control, and set on the devices that need it using a control as well).

Cases where another device in the pipeline would need to know the exact metadata format are exceedingly rare. I actually don't have any I could name right now.

Quoted 90 lines

If we come up with a good solution for metadata (and I hope we will), maybe we'll be able to use a similar mechanism for CFA patterns, simplifying new drivers and userspace. Or maybe this will remain a pipe dream given the backward compatibility implications.

Quoted 24 lines
Quoted 11 lines

The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).

These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.

One thing that may be worth considering or clarifying - for the case of the BCM2835 Unicam CSI-2 device, we only have 2x DMA output channels. So one will match image data packets, and the other will match "everything else". Typically "everything else" would only be CSI-2 embedded data, but in the case of the Raspberry Pi Camera v3 (IMX708), it includes embedded data, PDAF data, and HDR histogram data. Each of these outputs can be programmed to use a different packet ID in the sensor, but since Unicam only has a single DMA for "everything else", it all gets dumped into one metadata buffer. But given we know the exact structure of the data streams, it's trivial for useland to find the right bits in this buffer. Of course, other CSI-2 receivers with more DMA channels might allow these streams to end up in their own buffers.

Quoted 4 lines

Nothing in your series seems to stop us operating Unicam in this way, particularly because there is no fixed definition of the data format for V4L2_META_FMT_GENERIC_8. So I don't think it's a problem, but perhaps it's worth documenting that the metadata might include multiple streams from the sensor?

Thanks for your feedback Naush. Would you consider reviewing the individual patches in the series ? :-)

Quoted 52 lines
Quoted 52 lines

The internal (source) pads are an alternative to source routes [1]. The source routes were not universally liked and I do have to say I like re-using existing interface concepts (pads and everything you can do with pads, including access format, selections etc.) wherever it makes sense, instead of duplicating functionality.

Effectively internal source pads behave mostly just like sink pads, but they describe a flow of data that originates from a sub-device instead of arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable and disable routes from internal source pads to sub-device's source pads. The subdev format IOCTLs are usable, too, so one can find which subdev format is available on given internal source pad.

This set depends on these patches:

<URL:https://lore.kernel.org/linux-media/[email protected]/T/#t>

I've also pushed these here and I'll keep updating the branch:

<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>

Questions and comments are most welcome.

[1] <URL:https://lore.kernel.org/linux-media/[email protected]/>

Sakari Ailus (7):
media: mc: Add INTERNAL_SOURCE pad type flag
media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs
media: uapi: v4l: Document source routes
media: mc: Check pad flag validity
media: uapi: Add generic serial metadata mbus formats
media: uapi: Add generic 8-bit metadata format definitions
media: v4l: Support line-based metadata capture

 .../media/mediactl/media-types.rst            |   7 +
 .../userspace-api/media/v4l/dev-meta.rst      |  15 +
 .../userspace-api/media/v4l/dev-subdev.rst    |  18 +
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 317 ++++++++++++++++++
 .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++
 .../media/v4l/vidioc-enum-fmt.rst             |   7 +
 .../media/v4l/vidioc-subdev-g-routing.rst     |   5 +
 drivers/media/mc/mc-entity.c                  |  20 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
 drivers/media/v4l2-core/v4l2-subdev.c         |   6 +-
 include/uapi/linux/media-bus-format.h         |   9 +
 include/uapi/linux/media.h                    |   1 +
 include/uapi/linux/v4l2-subdev.h              |   6 +-
 include/uapi/linux/videodev2.h                |  19 ++
 15 files changed, 691 insertions(+), 5 deletions(-)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
Signature · 3 lines
Kind regards,

Sakari Ailus
#52 lore raw
In reply to Sakari Ailus

Hi Sakari

Sorry to be late to the party.

Quoted 60 lines · Sakari Ailus

Hi folks,

Here are a few patches to add support generic, line based metadata as well as internal source pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.

Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.

Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc. The number of these formats is probably going to be about as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.

Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.

Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).

The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).

These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.

The internal (source) pads are an alternative to source routes [1]. The source routes were not universally liked and I do have to say I like re-using existing interface concepts (pads and everything you can do with pads, including access format, selections etc.) wherever it makes sense, instead of duplicating functionality.

Quoted 6 lines

Effectively internal source pads behave mostly just like sink pads, but they describe a flow of data that originates from a sub-device instead of arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable and disable routes from internal source pads to sub-device's source pads. The subdev format IOCTLs are usable, too, so one can find which subdev format is available on given internal source pad.

Just to confirm my own understanding, the muxed streams API currently copes with situations such as the FPD-Link devices combining a single stream from each of a number of subdevs into one multiplexed stream using virtual channels, but it doesn't handle multiple data types from the same subdev. You're addressing that omission.

All seems reasonable.

One detail that I haven't seen covered and think ought to be clarified. For raw sensors where you have image data and metadata, in my experience the line width of that metadata and the packing format (8/10/12/14/16/20/24 bit) are all dictated by the image data configuration. You can't have eg 640 pixel wide _SBGGR10_1X10 image data and 320 pixel wide _META_1X8_8 metadata. (it has to be 640 pixel wide _META_1X10_10)

Which layer would be responsible for validating the configuration, and when?
a) The driver validates in set_stream, and fails if mismatched. That potentially results in lots of duplication between drivers.
b) set_pad_format on the image pad updates the formats and widths of the metadata pads automatically. Is there a way of notifying clients that the formats on potentially unrelated pads has changed, or is it something that just gets documented?
c) something else.

Either is workable, but IMHO it needs to be specified as to which is the expected behaviour.

Thanks
Dave

Quoted 43 lines

This set depends on these patches:

<URL:https://lore.kernel.org/linux-media/[email protected]/T/#t>

I've also pushed these here and I'll keep updating the branch:

<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>

Questions and comments are most welcome.

[1] <URL:https://lore.kernel.org/linux-media/[email protected]/>

Sakari Ailus (7):
media: mc: Add INTERNAL_SOURCE pad type flag
media: v4l: subdev: Support INTERNAL_SOURCE pads in routing IOCTLs
media: uapi: v4l: Document source routes
media: mc: Check pad flag validity
media: uapi: Add generic serial metadata mbus formats
media: uapi: Add generic 8-bit metadata format definitions
media: v4l: Support line-based metadata capture

 .../media/mediactl/media-types.rst            |   7 +
 .../userspace-api/media/v4l/dev-meta.rst      |  15 +
 .../userspace-api/media/v4l/dev-subdev.rst    |  18 +
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 317 ++++++++++++++++++
 .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++
 .../media/v4l/vidioc-enum-fmt.rst             |   7 +
 .../media/v4l/vidioc-subdev-g-routing.rst     |   5 +
 drivers/media/mc/mc-entity.c                  |  20 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
 drivers/media/v4l2-core/v4l2-subdev.c         |   6 +-
 include/uapi/linux/media-bus-format.h         |   9 +
 include/uapi/linux/media.h                    |   1 +
 include/uapi/linux/v4l2-subdev.h              |   6 +-
 include/uapi/linux/videodev2.h                |  19 ++
 15 files changed, 691 insertions(+), 5 deletions(-)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst

--
2.30.2

author maintainer
#53 lore raw

Hi Dave,

Quoted 72 lines · Dave Stevenson

Hi Sakari

Sorry to be late to the party.

Quoted 60 lines · Sakari Ailus

Hi folks,

Here are a few patches to add support generic, line based metadata as well as internal source pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.

Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.

Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc. The number of these formats is probably going to be about as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.

Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.

Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).

The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).

These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.

The internal (source) pads are an alternative to source routes [1]. The source routes were not universally liked and I do have to say I like re-using existing interface concepts (pads and everything you can do with pads, including access format, selections etc.) wherever it makes sense, instead of duplicating functionality.

Quoted 6 lines

Effectively internal source pads behave mostly just like sink pads, but they describe a flow of data that originates from a sub-device instead of arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable and disable routes from internal source pads to sub-device's source pads. The subdev format IOCTLs are usable, too, so one can find which subdev format is available on given internal source pad.

Quoted 5 lines

Just to confirm my own understanding, the muxed streams API currently copes with situations such as the FPD-Link devices combining a single stream from each of a number of subdevs into one multiplexed stream using virtual channels, but it doesn't handle multiple data types from the same subdev. You're addressing that omission.

Correct.

Quoted 10 lines

All seems reasonable.

One detail that I haven't seen covered and think ought to be clarified. For raw sensors where you have image data and metadata, in my experience the line width of that metadata and the packing format (8/10/12/14/16/20/24 bit) are all dictated by the image data configuration. You can't have eg 640 pixel wide _SBGGR10_1X10 image data and 320 pixel wide _META_1X8_8 metadata. (it has to be 640 pixel wide _META_1X10_10)

That corresponds to my experience and is probably related to sensor implementation rather than CSI-2 itself.

Quoted 10 lines

Which layer would be responsible for validating the configuration, and when?
a) The driver validates in set_stream, and fails if mismatched. That potentially results in lots of duplication between drivers.
b) set_pad_format on the image pad updates the formats and widths of the metadata pads automatically. Is there a way of notifying clients that the formats on potentially unrelated pads has changed, or is it something that just gets documented?
c) something else.

Quoted 2 lines

Either is workable, but IMHO it needs to be specified as to which is the expected behaviour.

I left that out of the spec as in the end this depends on the hardware constraints and how the driver author decided to map that to the API.

I expect sensor drivers to let the user configure the image format freely (as it's been so far) and if there's anything to configure in the metadata format that does not affect the image data format, then that configurability is available on the metadata internal sink pad via SUBDEV_S_FMT. So from the above options b appears a good fit. SUBDEV_S_FMT does not have notification ability and probably shouldn't, instead the order of configuration should be followed.

I think this would be good to be documented, perhaps with the first sensor driver that uses it. Likely to be merged with this set later on.

Signature · 3 lines
Kind regards,

Sakari Ailus
maintainer
#54 lore raw
Quoted 87 lines · Sakari Ailus
Quoted 68 lines · Dave Stevenson
Quoted 60 lines · Sakari Ailus

Hi folks,

Here are a few patches to add support generic, line based metadata as well as internal source pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.

Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.

Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc. The number of these formats is probably going to be about as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.

Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.

Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).

The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).

These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.

The internal (source) pads are an alternative to source routes [1]. The source routes were not universally liked and I do have to say I like re-using existing interface concepts (pads and everything you can do with pads, including access format, selections etc.) wherever it makes sense, instead of duplicating functionality.

Quoted 6 lines

Effectively internal source pads behave mostly just like sink pads, but they describe a flow of data that originates from a sub-device instead of arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable and disable routes from internal source pads to sub-device's source pads. The subdev format IOCTLs are usable, too, so one can find which subdev format is available on given internal source pad.

Quoted 5 lines

Just to confirm my own understanding, the muxed streams API currently copes with situations such as the FPD-Link devices combining a single stream from each of a number of subdevs into one multiplexed stream using virtual channels, but it doesn't handle multiple data types from the same subdev. You're addressing that omission.

Correct.

Quoted 9 lines

All seems reasonable.

One detail that I haven't seen covered and think ought to be clarified. For raw sensors where you have image data and metadata, in my experience the line width of that metadata and the packing format (8/10/12/14/16/20/24 bit) are all dictated by the image data configuration. You can't have eg 640 pixel wide _SBGGR10_1X10 image data and 320 pixel wide _META_1X8_8 metadata. (it has to be 640 pixel wide _META_1X10_10)

Quoted 2 lines

That corresponds to my experience and is probably related to sensor implementation rather than CSI-2 itself.

I agree, in theory the above could be implemented by a CSI-2 source, but in practice that would be very uncommon.

Quoted 23 lines
Quoted 11 lines

Which layer would be responsible for validating the configuration, and when?
a) The driver validates in set_stream, and fails if mismatched. That potentially results in lots of duplication between drivers.
b) set_pad_format on the image pad updates the formats and widths of the metadata pads automatically. Is there a way of notifying clients that the formats on potentially unrelated pads has changed, or is it something that just gets documented?
c) something else.

Either is workable, but IMHO it needs to be specified as to which is the expected behaviour.

I left that out of the spec as in the end this depends on the hardware constraints and how the driver author decided to map that to the API.

I expect sensor drivers to let the user configure the image format freely (as it's been so far) and if there's anything to configure in the metadata format that does not affect the image data format, then that configurability is available on the metadata internal sink pad via SUBDEV_S_FMT. So from the above options b appears a good fit. SUBDEV_S_FMT does not have notification ability and probably shouldn't, instead the order of configuration should be followed.

Quoted 2 lines

I think this would be good to be documented, perhaps with the first sensor driver that uses it. Likely to be merged with this set later on.

I agree with the above. The behaviour depends on the hardware constraints, and is thus device-dependent, but we can standardize it for classes of devices. For raw sensors that produce image and embedded data streams, option B should be required in the documentation.

Signature · 3 lines
Regards,

Laurent Pinchart