Skip to contentSkip to conversation

Generic line based metadata support, internal pads

PATCH Superseded Sakari Ailus 5 participants 84 messages Aug 8, 2023 – Sep 22, 2023

Subsystems (2)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Sakari Ailus

author maintainer
#0 lore raw

v3 cover Generic line based metadata support, internal pads

Hi folks,

Here are a few patches to add support generic, line based metadata as well as internal pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.

Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.

Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc. The number of these formats is probably going to be about as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.

Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.

Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).

The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).

These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.

The internal (source) pads are an alternative to source routes [1]. The source routes were not universally liked and I do have to say I like re-using existing interface concepts (pads and everything you can do with pads, including access format, selections etc.) wherever it makes sense, instead of duplicating functionality.

Effectively internal source pads behave mostly just like sink pads, but they describe a flow of data that originates from a sub-device instead of arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable and disable routes from internal source pads to sub-device's source pads. The subdev format IOCTLs are usable, too, so one can find which subdev format is available on given internal source pad.

This set depends on these patches:

<URL:https://lore.kernel.org/linux-media/[email protected]/T/#t>

I've also pushed these here and I'll keep updating the branch:

<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>

Questions and comments are most welcome.

Driver support is to be added, as well as an example.

[1] <URL:https://lore.kernel.org/linux-media/[email protected]/>

since v3:

- Add a better example, with formats.

- Add CCS static data media bus codes.

since v2:

- Added an example demonstrating the use of internal pads. --- Is the level of detail enough for the purpose?

- Improved documentation.

- Added a macro to tell whether a format is a metadata format. (Documentation could be added.)

- A small ReST syntax fix in the same section.

- Drop leftovers of a patch checking for the INTERNAL_SOURCE flag.

since v1:

- Make the new pad flag just "INTERNAL", requiring either SINK or SOURCE pad flag to accompany it. Removed the union in struct v4l2_subdev_route.

- Add the term "stream" to MC glossary.

- Improved and fixed documentation (according to comments).

- Note these formats are little endian.

- Remove 1X8 from the names of the mbus codes. These formats have generally 8 bits per pixel.

- Fix mbus code numbering (had holes in RFC).

- Add new metadata fields to debug prints.

- Fix a minor documentation build issue.

Sakari Ailus (10):
media: Documentation: Align numbered list
media: mc: Check pad flag validity
media: mc: Add INTERNAL pad flag
media: uapi: Add generic serial metadata mbus formats
media: uapi: Document which mbus format fields are valid for metadata
media: uapi: Add a macro to tell whether an mbus code is metadata
media: uapi: Add generic 8-bit metadata format definitions
media: v4l: Support line-based metadata capture
media: Add media bus codes for MIPI CCS embedded data
media: uapi: v4l: Document source routes

 .../userspace-api/media/glossary.rst          |   6 +
 .../media/mediactl/media-types.rst            |   6 +
 .../userspace-api/media/v4l/dev-meta.rst      |  15 +
 .../userspace-api/media/v4l/dev-subdev.rst    | 206 ++++++++++-
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 331 ++++++++++++++++++
 .../media/v4l/subdev-formats.rst              | 289 +++++++++++++++
 .../media/v4l/vidioc-enum-fmt.rst             |   7 +
 .../media/videodev2.h.rst.exceptions          |   1 +
 drivers/media/mc/mc-entity.c                  |  17 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          |  13 +-
 include/uapi/linux/media-bus-format.h         |  20 ++
 include/uapi/linux/media.h                    |   1 +
 include/uapi/linux/v4l2-mediabus.h            |  18 +-
 include/uapi/linux/videodev2.h                |  19 +
 15 files changed, 926 insertions(+), 24 deletions(-)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
Signature · 1 line
2.39.2
author maintainer
#1 lore raw

PATCH v3 1/10 media: Documentation: Align numbered list

Align lines for numbered list so that Sphinx produces an uniform output for all list entries.

---
 .../userspace-api/media/v4l/dev-subdev.rst    | 27 +++++++++----------
 1 file changed, 13 insertions(+), 14 deletions(-)
Documentation/userspace-api/media/v4l/dev-subdev.rst +13 -14
index a4f1df7093e8..b023918177b5 100644
@@ -579,20 +579,19 @@ is started.
 
 There are three steps in configuring the streams:
 
-1) Set up links. Connect the pads between sub-devices using the :ref:`Media
-Controller API <media_controller>`
+1) Set up links. Connect the pads between sub-devices using the
+   :ref:`Media Controller API <media_controller>`
 
-2) Streams. Streams are declared and their routing is configured by
-setting the routing table for the sub-device using
-:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that
-setting the routing table will reset formats and selections in the
-sub-device to default values.
+2) Streams. Streams are declared and their routing is configured by setting the
+   routing table for the sub-device using :ref:`VIDIOC_SUBDEV_S_ROUTING
+   <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that setting the routing table will
+   reset formats and selections in the sub-device to default values.
 
-3) Configure formats and selections. Formats and selections of each stream
-are configured separately as documented for plain sub-devices in
-:ref:`format-propagation`. The stream ID is set to the same stream ID
-associated with either sink or source pads of routes configured using the
-:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
+3) Configure formats and selections. Formats and selections of each stream are
+   configured separately as documented for plain sub-devices in
+   :ref:`format-propagation`. The stream ID is set to the same stream ID
+   associated with either sink or source pads of routes configured using the
+   :ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
 
 Multiplexed streams setup example
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -619,8 +618,8 @@ modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
 To configure this pipeline, the userspace must take the following steps:
 
 1) Set up media links between entities: connect the sensors to the bridge,
-bridge to the receiver, and the receiver to the DMA engines. This step does
-not differ from normal non-multiplexed media controller setup.
+   bridge to the receiver, and the receiver to the DMA engines. This step does
+   not differ from normal non-multiplexed media controller setup.
 
 2) Configure routing
 
Signature · 1 line
2.39.2
maintainer
#2 lore raw

Re: [PATCH v3 01/10] media: Documentation: Align numbered list

Hi Sakari,

Thank you for the patch.

Quoted 13 lines · Sakari Ailus

Align lines for numbered list so that Sphinx produces an uniform output for all list entries.

---
 .../userspace-api/media/v4l/dev-subdev.rst    | 27 +++++++++----------
 1 file changed, 13 insertions(+), 14 deletions(-)
Documentation/userspace-api/media/v4l/dev-subdev.rst +0 -0
index a4f1df7093e8..b023918177b5 100644

Documentation/userspace-api/media/v4l/dev-subdev.rst is started. · line 583

Quoted 7 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +2 -2
@@ -579,20 +579,19 @@ is started.
 
 There are three steps in configuring the streams:
 
-1) Set up links. Connect the pads between sub-devices using the :ref:`Media
-Controller API <media_controller>`
+1) Set up links. Connect the pads between sub-devices using the
+   :ref:`Media Controller API <media_controller>`

While at it, could we standardize on 1. vs. 1) ? Both are used in the same file. After this patch they both render to "1.", but it would be nice to be consistent in the source too.

Quoted 25 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +9 -10
-2) Streams. Streams are declared and their routing is configured by
-setting the routing table for the sub-device using
-:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that
-setting the routing table will reset formats and selections in the
-sub-device to default values.
+2) Streams. Streams are declared and their routing is configured by setting the
+   routing table for the sub-device using :ref:`VIDIOC_SUBDEV_S_ROUTING
+   <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that setting the routing table will
+   reset formats and selections in the sub-device to default values.
 
-3) Configure formats and selections. Formats and selections of each stream
-are configured separately as documented for plain sub-devices in
-:ref:`format-propagation`. The stream ID is set to the same stream ID
-associated with either sink or source pads of routes configured using the
-:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
+3) Configure formats and selections. Formats and selections of each stream are
+   configured separately as documented for plain sub-devices in
+   :ref:`format-propagation`. The stream ID is set to the same stream ID
+   associated with either sink or source pads of routes configured using the
+   :ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
 
 Multiplexed streams setup example
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Documentation/userspace-api/media/v4l/dev-subdev.rst modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes. · line 625

Quoted 10 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +2 -2
@@ -619,8 +618,8 @@ modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
 To configure this pipeline, the userspace must take the following steps:
 
 1) Set up media links between entities: connect the sensors to the bridge,
-bridge to the receiver, and the receiver to the DMA engines. This step does
-not differ from normal non-multiplexed media controller setup.
+   bridge to the receiver, and the receiver to the DMA engines. This step does
+   not differ from normal non-multiplexed media controller setup.
 
 2) Configure routing
 

3) below contains two paragraphs that are not indented.

Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#3 lore raw

Re: [PATCH v3 01/10] media: Documentation: Align numbered list

Hi Laurent,

Quoted 21 lines · Laurent Pinchart

Hi Sakari,

Thank you for the patch.

Quoted 16 lines · Sakari Ailus

Align lines for numbered list so that Sphinx produces an uniform output for all list entries.

---
 .../userspace-api/media/v4l/dev-subdev.rst    | 27 +++++++++----------
 1 file changed, 13 insertions(+), 14 deletions(-)
Documentation/userspace-api/media/v4l/dev-subdev.rst +0 -0
index a4f1df7093e8..b023918177b5 100644
@@ -579,20 +579,19 @@ is started.
 
 There are three steps in configuring the streams:
 
Quoted 8 lines

Documentation/userspace-api/media/v4l/dev-subdev.rst is started. · line 583

Quoted 4 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +2 -2
-1) Set up links. Connect the pads between sub-devices using the :ref:`Media
-Controller API <media_controller>`
+1) Set up links. Connect the pads between sub-devices using the
+   :ref:`Media Controller API <media_controller>`

While at it, could we standardize on 1. vs. 1) ? Both are used in the same file. After this patch they both render to "1.", but it would be nice to be consistent in the source too.

I'll check the rest of the file. The number followed by a trailing dot is a numbered list in ReST, we should align with that.

Quoted 34 lines
Quoted 34 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +11 -12
-2) Streams. Streams are declared and their routing is configured by
-setting the routing table for the sub-device using
-:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that
-setting the routing table will reset formats and selections in the
-sub-device to default values.
+2) Streams. Streams are declared and their routing is configured by setting the
+   routing table for the sub-device using :ref:`VIDIOC_SUBDEV_S_ROUTING
+   <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that setting the routing table will
+   reset formats and selections in the sub-device to default values.
 
-3) Configure formats and selections. Formats and selections of each stream
-are configured separately as documented for plain sub-devices in
-:ref:`format-propagation`. The stream ID is set to the same stream ID
-associated with either sink or source pads of routes configured using the
-:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
+3) Configure formats and selections. Formats and selections of each stream are
+   configured separately as documented for plain sub-devices in
+   :ref:`format-propagation`. The stream ID is set to the same stream ID
+   associated with either sink or source pads of routes configured using the
+   :ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
 
 Multiplexed streams setup example
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -619,8 +618,8 @@ modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
 To configure this pipeline, the userspace must take the following steps:
 
 1) Set up media links between entities: connect the sensors to the bridge,
-bridge to the receiver, and the receiver to the DMA engines. This step does
-not differ from normal non-multiplexed media controller setup.
+   bridge to the receiver, and the receiver to the DMA engines. This step does
+   not differ from normal non-multiplexed media controller setup.
 
Quoted 4 lines

Documentation/userspace-api/media/v4l/dev-subdev.rst modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes. · line 625

Quoted 2 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +0 -0
 2) Configure routing
 

3) below contains two paragraphs that are not indented.

From the documentation output this does not matter. Or was your comment about aligning the formatting in the source? (I prefer it as-is but no strong opinion about it either. Just slightly easier to edit that way. :-))

Signature · 3 lines
Regards,

Sakari Ailus
maintainer
#4 lore raw

Re: [PATCH v3 01/10] media: Documentation: Align numbered list

Quoted 64 lines · Sakari Ailus
Quoted 17 lines · Laurent Pinchart
Quoted 16 lines · Sakari Ailus

Align lines for numbered list so that Sphinx produces an uniform output for all list entries.

---
 .../userspace-api/media/v4l/dev-subdev.rst    | 27 +++++++++----------
 1 file changed, 13 insertions(+), 14 deletions(-)
Documentation/userspace-api/media/v4l/dev-subdev.rst +0 -0
index a4f1df7093e8..b023918177b5 100644
@@ -579,20 +579,19 @@ is started.
 
 There are three steps in configuring the streams:
 
Quoted 8 lines

Documentation/userspace-api/media/v4l/dev-subdev.rst is started. · line 583

Quoted 4 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +2 -2
-1) Set up links. Connect the pads between sub-devices using the :ref:`Media
-Controller API <media_controller>`
+1) Set up links. Connect the pads between sub-devices using the
+   :ref:`Media Controller API <media_controller>`

While at it, could we standardize on 1. vs. 1) ? Both are used in the same file. After this patch they both render to "1.", but it would be nice to be consistent in the source too.

I'll check the rest of the file. The number followed by a trailing dot is a numbered list in ReST, we should align with that.

Quoted 34 lines
Quoted 34 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +11 -12
-2) Streams. Streams are declared and their routing is configured by
-setting the routing table for the sub-device using
-:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that
-setting the routing table will reset formats and selections in the
-sub-device to default values.
+2) Streams. Streams are declared and their routing is configured by setting the
+   routing table for the sub-device using :ref:`VIDIOC_SUBDEV_S_ROUTING
+   <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that setting the routing table will
+   reset formats and selections in the sub-device to default values.
 
-3) Configure formats and selections. Formats and selections of each stream
-are configured separately as documented for plain sub-devices in
-:ref:`format-propagation`. The stream ID is set to the same stream ID
-associated with either sink or source pads of routes configured using the
-:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
+3) Configure formats and selections. Formats and selections of each stream are
+   configured separately as documented for plain sub-devices in
+   :ref:`format-propagation`. The stream ID is set to the same stream ID
+   associated with either sink or source pads of routes configured using the
+   :ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
 
 Multiplexed streams setup example
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -619,8 +618,8 @@ modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
 To configure this pipeline, the userspace must take the following steps:
 
 1) Set up media links between entities: connect the sensors to the bridge,
-bridge to the receiver, and the receiver to the DMA engines. This step does
-not differ from normal non-multiplexed media controller setup.
+   bridge to the receiver, and the receiver to the DMA engines. This step does
+   not differ from normal non-multiplexed media controller setup.
 
Quoted 8 lines
Quoted 4 lines

Documentation/userspace-api/media/v4l/dev-subdev.rst modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes. · line 625

Quoted 2 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +0 -0
 2) Configure routing
 

3) below contains two paragraphs that are not indented.

From the documentation output this does not matter. Or was your comment about aligning the formatting in the source? (I prefer it as-is but no strong opinion about it either. Just slightly easier to edit that way. :-))

I like consistency, both in the source and the output :-)

Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#5 lore raw

Re: [PATCH v3 01/10] media: Documentation: Align numbered list

Quoted 38 lines · Laurent Pinchart
Quoted 37 lines · Sakari Ailus
Quoted 36 lines
Quoted 24 lines

-2) Streams. Streams are declared and their routing is configured by

Documentation/userspace-api/media/v4l/dev-subdev.rst +9 -9
-setting the routing table for the sub-device using
-:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that
-setting the routing table will reset formats and selections in the
-sub-device to default values.
+2) Streams. Streams are declared and their routing is configured by setting the
+   routing table for the sub-device using :ref:`VIDIOC_SUBDEV_S_ROUTING
+   <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that setting the routing table will
+   reset formats and selections in the sub-device to default values.
 
-3) Configure formats and selections. Formats and selections of each stream
-are configured separately as documented for plain sub-devices in
-:ref:`format-propagation`. The stream ID is set to the same stream ID
-associated with either sink or source pads of routes configured using the
-:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
+3) Configure formats and selections. Formats and selections of each stream are
+   configured separately as documented for plain sub-devices in
+   :ref:`format-propagation`. The stream ID is set to the same stream ID
+   associated with either sink or source pads of routes configured using the
+   :ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
 
 Multiplexed streams setup example
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Documentation/userspace-api/media/v4l/dev-subdev.rst modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes. · line 625

Quoted 10 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +2 -2
@@ -619,8 +618,8 @@ modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
 To configure this pipeline, the userspace must take the following steps:
 
 1) Set up media links between entities: connect the sensors to the bridge,
-bridge to the receiver, and the receiver to the DMA engines. This step does
-not differ from normal non-multiplexed media controller setup.
+   bridge to the receiver, and the receiver to the DMA engines. This step does
+   not differ from normal non-multiplexed media controller setup.
 
 2) Configure routing
 

3) below contains two paragraphs that are not indented.

Quoted 5 lines
Quoted 3 lines

From the documentation output this does not matter. Or was your comment about aligning the formatting in the source? (I prefer it as-is but no strong opinion about it either. Just slightly easier to edit that way. :-))

I like consistency, both in the source and the output :-)

This doesn't seem to be possible in all cases --- Sphinx thinks the paragraph is part of the preceding table if it's indented.

Signature · 1 line
Sakari Ailus
author maintainer
#6 lore raw

PATCH v3 2/10 media: mc: Check pad flag validity

In reply to Sakari Ailus

Check the validity of pad flags on entity init. Exactly one of the flags must be set.

---
 drivers/media/mc/mc-entity.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
drivers/media/mc/mc-entity.c +10 -1
index 83468d4a440b..4991281dcccc 100644
@@ -197,6 +197,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	struct media_device *mdev = entity->graph_obj.mdev;
 	struct media_pad *iter;
 	unsigned int i = 0;
+	int ret = 0;
 
 	if (num_pads >= MEDIA_ENTITY_MAX_PADS)
 		return -E2BIG;
@@ -210,6 +211,14 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	media_entity_for_each_pad(entity, iter) {
 		iter->entity = entity;
 		iter->index = i++;
+
+		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
+					     MEDIA_PAD_FL_SOURCE))
+		    != 1) {
+			ret = -EINVAL;
+			break;
+		}
+
 		if (mdev)
 			media_gobj_create(mdev, MEDIA_GRAPH_PAD,
 					  &iter->graph_obj);
@@ -218,7 +227,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	if (mdev)
 		mutex_unlock(&mdev->graph_mutex);
 
-	return 0;
+	return ret;
 }
 EXPORT_SYMBOL_GPL(media_entity_pads_init);
 
Signature · 1 line
2.39.2
#7 lore raw

Re: [PATCH v3 02/10] media: mc: Check pad flag validity

Hi Sakari

Quoted 21 lines · Sakari Ailus

Check the validity of pad flags on entity init. Exactly one of the flags must be set.

---
 drivers/media/mc/mc-entity.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
drivers/media/mc/mc-entity.c +1 -0
index 83468d4a440b..4991281dcccc 100644
@@ -197,6 +197,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	struct media_device *mdev = entity->graph_obj.mdev;
 	struct media_pad *iter;
 	unsigned int i = 0;
+	int ret = 0;
 
 	if (num_pads >= MEDIA_ENTITY_MAX_PADS)
 		return -E2BIG;

drivers/media/mc/mc-entity.c int media_entity_pads_init(struct media_entity *entity, u16 num_pads, · line 217

Quoted 7 lines
drivers/media/mc/mc-entity.c +4 -0
@@ -210,6 +211,14 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	media_entity_for_each_pad(entity, iter) {
 		iter->entity = entity;
 		iter->index = i++;
+
+		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
+					     MEDIA_PAD_FL_SOURCE))
+		    != 1) {

Fits on the previous line, if you like it

drivers/media/mc/mc-entity.c int media_entity_pads_init(struct media_entity *entity, u16 num_pads, · line 219

Quoted 2 lines
drivers/media/mc/mc-entity.c +2 -0
+			ret = -EINVAL;
+			break;

Now that we can interrupt the for loop earlier, do you need to cleanup objects appended to the mdev list with the media_gobj_create() call here below ?

Quoted 17 lines
drivers/media/mc/mc-entity.c +3 -2
+		}
+
 		if (mdev)
 			media_gobj_create(mdev, MEDIA_GRAPH_PAD,
 					  &iter->graph_obj);
@@ -218,7 +227,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	if (mdev)
 		mutex_unlock(&mdev->graph_mutex);
 
-	return 0;
+	return ret;
 }
 EXPORT_SYMBOL_GPL(media_entity_pads_init);
 
--

2.39.2

author maintainer
#8 lore raw

Re: [PATCH v3 02/10] media: mc: Check pad flag validity

Hi Jacopo,

Quoted 21 lines · Jacopo Mondi

Hi Sakari

Quoted 18 lines · Sakari Ailus

Check the validity of pad flags on entity init. Exactly one of the flags must be set.

---
 drivers/media/mc/mc-entity.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
drivers/media/mc/mc-entity.c +1 -0
index 83468d4a440b..4991281dcccc 100644
@@ -197,6 +197,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	struct media_device *mdev = entity->graph_obj.mdev;
 	struct media_pad *iter;
 	unsigned int i = 0;
+	int ret = 0;
 
Quoted 12 lines

drivers/media/mc/mc-entity.c int media_entity_pads_init(struct media_entity *entity, u16 num_pads, · line 217

Quoted 10 lines
drivers/media/mc/mc-entity.c +4 -0
 	if (num_pads >= MEDIA_ENTITY_MAX_PADS)
 		return -E2BIG;
@@ -210,6 +211,14 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	media_entity_for_each_pad(entity, iter) {
 		iter->entity = entity;
 		iter->index = i++;
+
+		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
+					     MEDIA_PAD_FL_SOURCE))
+		    != 1) {

Fits on the previous line, if you like it

Not quite.

Quoted 6 lines

drivers/media/mc/mc-entity.c int media_entity_pads_init(struct media_entity *entity, u16 num_pads, · line 219

Quoted 3 lines
drivers/media/mc/mc-entity.c +2 -0
+			ret = -EINVAL;
+			break;

Now that we can interrupt the for loop earlier, do you need to cleanup objects appended to the mdev list with the media_gobj_create() call here below ?

Nice find. I'll fix this for v4.

Quoted 15 lines
Quoted 15 lines
drivers/media/mc/mc-entity.c +3 -1
+		}
+
 		if (mdev)
 			media_gobj_create(mdev, MEDIA_GRAPH_PAD,
 					  &iter->graph_obj);
@@ -218,7 +227,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	if (mdev)
 		mutex_unlock(&mdev->graph_mutex);
 
-	return 0;
+	return ret;
 }
 EXPORT_SYMBOL_GPL(media_entity_pads_init);
 
Signature · 3 lines
Kind regards,

Sakari Ailus
maintainer
#9 lore raw

Re: [PATCH v3 02/10] media: mc: Check pad flag validity

Quoted 33 lines · Sakari Ailus
Quoted 19 lines · Jacopo Mondi
Quoted 18 lines · Sakari Ailus

Check the validity of pad flags on entity init. Exactly one of the flags must be set.

---
 drivers/media/mc/mc-entity.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
drivers/media/mc/mc-entity.c +1 -0
index 83468d4a440b..4991281dcccc 100644
@@ -197,6 +197,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	struct media_device *mdev = entity->graph_obj.mdev;
 	struct media_pad *iter;
 	unsigned int i = 0;
+	int ret = 0;
 
Quoted 12 lines

drivers/media/mc/mc-entity.c int media_entity_pads_init(struct media_entity *entity, u16 num_pads, · line 217

Quoted 10 lines
drivers/media/mc/mc-entity.c +4 -0
 	if (num_pads >= MEDIA_ENTITY_MAX_PADS)
 		return -E2BIG;
@@ -210,6 +211,14 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	media_entity_for_each_pad(entity, iter) {
 		iter->entity = entity;
 		iter->index = i++;
+
+		if (hweight32(iter->flags & (MEDIA_PAD_FL_SINK |
+					     MEDIA_PAD_FL_SOURCE))
+		    != 1) {

Fits on the previous line, if you like it

Quoted 1 line

Not quite.

The '!= 1) {' does. unless you want to lower the line length limit to 73 or less :-)

Quoted 22 lines
Quoted 5 lines

drivers/media/mc/mc-entity.c int media_entity_pads_init(struct media_entity *entity, u16 num_pads, · line 219

Quoted 2 lines
drivers/media/mc/mc-entity.c +2 -0
+			ret = -EINVAL;
+			break;

Now that we can interrupt the for loop earlier, do you need to cleanup objects appended to the mdev list with the media_gobj_create() call here below ?

Nice find. I'll fix this for v4.

Quoted 14 lines
Quoted 14 lines
drivers/media/mc/mc-entity.c +3 -1
+		}
+
 		if (mdev)
 			media_gobj_create(mdev, MEDIA_GRAPH_PAD,
 					  &iter->graph_obj);
@@ -218,7 +227,7 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	if (mdev)
 		mutex_unlock(&mdev->graph_mutex);
 
-	return 0;
+	return ret;
 }
 EXPORT_SYMBOL_GPL(media_entity_pads_init);
 
Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#10 lore raw

PATCH v3 3/10 media: mc: Add INTERNAL pad flag

In reply to Sakari Ailus

Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.

Also prevent creating links to pads that have been flagged as internal.

---
 Documentation/userspace-api/media/glossary.rst             | 6 ++++++
 Documentation/userspace-api/media/mediactl/media-types.rst | 6 ++++++
 drivers/media/mc/mc-entity.c                               | 6 +++++-
 include/uapi/linux/media.h                                 | 1 +
 4 files changed, 18 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/glossary.rst +6 -0
index 96a360edbf3b..f7b99a4527c7 100644
@@ -173,6 +173,12 @@ Glossary
 	An integrated circuit that integrates all components of a computer
 	or other electronic systems.
 
+_media-glossary-stream:
+    Stream
+	A distinct flow of data (image data or metadata) over a media pipeline
+	from source to sink. A source may be e.g. an image sensor and a sink
+	e.g. a memory buffer.
+
     V4L2 API
 	**V4L2 userspace API**
 
Documentation/userspace-api/media/mediactl/media-types.rst +6 -0
index 0ffeece1e0c8..28941da27790 100644
@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
 .. _MEDIA-PAD-FL-SINK:
 .. _MEDIA-PAD-FL-SOURCE:
 .. _MEDIA-PAD-FL-MUST-CONNECT:
+.. _MEDIA-PAD-FL-INTERNAL:
 
 .. flat-table:: Media pad flags
     :header-rows:  0
@@ -382,6 +383,11 @@ Types and flags used to represent the media graph elements
 	  when this flag isn't set; the absence of the flag doesn't imply
 	  there is none.
 
+    *  -  ``MEDIA_PAD_FL_INTERNAL``
+       -  The internal flag indicates an internal pad that has no external
+	  connections. Such a pad shall not be connected with a link. The
+	  internal flag indicates that the :ref:``stream
+	  <media-glossary-stream>`` either starts or ends in the entity.
 
 One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
 must be set for every pad.
drivers/media/mc/mc-entity.c +5 -1
index 4991281dcccc..03a9e0b8ebab 100644
@@ -1071,7 +1071,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
 
 	for (i = 0; i < entity->num_pads; i++) {
 		if ((entity->pads[i].flags &
-		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
+		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
+		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
 			continue;
 
 		if (entity->pads[i].sig_type == sig_type)
@@ -1094,6 +1095,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
 		return -EINVAL;
 	if (WARN_ON(!(source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE)))
 		return -EINVAL;
+	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE &&
+		    source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL))
+		return -EINVAL;
 	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
 		return -EINVAL;
 
include/uapi/linux/media.h +1 -0
index 1c80b1d6bbaf..80cfd12a43fc 100644
@@ -208,6 +208,7 @@ struct media_entity_desc {
 #define MEDIA_PAD_FL_SINK			(1U << 0)
 #define MEDIA_PAD_FL_SOURCE			(1U << 1)
 #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
+#define MEDIA_PAD_FL_INTERNAL			(1U << 3)
 
 struct media_pad_desc {
 	__u32 entity;		/* entity ID */
Signature · 1 line
2.39.2
#11 lore raw

Re: [PATCH v3 03/10] media: mc: Add INTERNAL pad flag

Quoted 69 lines · Sakari Ailus

Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.

Also prevent creating links to pads that have been flagged as internal.

---
 Documentation/userspace-api/media/glossary.rst             | 6 ++++++
 Documentation/userspace-api/media/mediactl/media-types.rst | 6 ++++++
 drivers/media/mc/mc-entity.c                               | 6 +++++-
 include/uapi/linux/media.h                                 | 1 +
 4 files changed, 18 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/glossary.rst +6 -0
index 96a360edbf3b..f7b99a4527c7 100644
@@ -173,6 +173,12 @@ Glossary
 	An integrated circuit that integrates all components of a computer
 	or other electronic systems.
 
+_media-glossary-stream:
+    Stream
+	A distinct flow of data (image data or metadata) over a media pipeline
+	from source to sink. A source may be e.g. an image sensor and a sink
+	e.g. a memory buffer.
+
     V4L2 API
 	**V4L2 userspace API**
 
Documentation/userspace-api/media/mediactl/media-types.rst +6 -0
index 0ffeece1e0c8..28941da27790 100644
@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
 .. _MEDIA-PAD-FL-SINK:
 .. _MEDIA-PAD-FL-SOURCE:
 .. _MEDIA-PAD-FL-MUST-CONNECT:
+.. _MEDIA-PAD-FL-INTERNAL:
 
 .. flat-table:: Media pad flags
     :header-rows:  0
@@ -382,6 +383,11 @@ Types and flags used to represent the media graph elements
 	  when this flag isn't set; the absence of the flag doesn't imply
 	  there is none.
 
+    *  -  ``MEDIA_PAD_FL_INTERNAL``
+       -  The internal flag indicates an internal pad that has no external
+	  connections. Such a pad shall not be connected with a link. The
+	  internal flag indicates that the :ref:``stream
+	  <media-glossary-stream>`` either starts or ends in the entity.
 
 One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
 must be set for every pad.
drivers/media/mc/mc-entity.c +2 -1
index 4991281dcccc..03a9e0b8ebab 100644
@@ -1071,7 +1071,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
 
 	for (i = 0; i < entity->num_pads; i++) {
 		if ((entity->pads[i].flags &
-		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
+		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
+		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
 			continue;
 
 		if (entity->pads[i].sig_type == sig_type)

drivers/media/mc/mc-entity.c media_create_pad_link(struct media_entity *source, u16 source_pad, · line 1099

Quoted 5 lines
drivers/media/mc/mc-entity.c +2 -0
@@ -1094,6 +1095,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
 		return -EINVAL;
 	if (WARN_ON(!(source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE)))
 		return -EINVAL;
+	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE &&
+		    source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL))

Even though this code is correct, I prefer to have () around the 'x & y' parts. It avoids confusion for the reader.

Up to you, though.

Regards,

Hans

Quoted 16 lines
drivers/media/mc/mc-entity.c +1 -0
+		return -EINVAL;
 	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
 		return -EINVAL;
 
include/uapi/linux/media.h +1 -0
index 1c80b1d6bbaf..80cfd12a43fc 100644
@@ -208,6 +208,7 @@ struct media_entity_desc {
 #define MEDIA_PAD_FL_SINK			(1U << 0)
 #define MEDIA_PAD_FL_SOURCE			(1U << 1)
 #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
+#define MEDIA_PAD_FL_INTERNAL			(1U << 3)
 
 struct media_pad_desc {
 	__u32 entity;		/* entity ID */
author maintainer
#12 lore raw

Re: [PATCH v3 03/10] media: mc: Add INTERNAL pad flag

Hi Hans,

Quoted 68 lines · Hans Verkuil
Quoted 67 lines · Sakari Ailus

Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.

Also prevent creating links to pads that have been flagged as internal.

---
 Documentation/userspace-api/media/glossary.rst             | 6 ++++++
 Documentation/userspace-api/media/mediactl/media-types.rst | 6 ++++++
 drivers/media/mc/mc-entity.c                               | 6 +++++-
 include/uapi/linux/media.h                                 | 1 +
 4 files changed, 18 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/glossary.rst +6 -0
index 96a360edbf3b..f7b99a4527c7 100644
@@ -173,6 +173,12 @@ Glossary
 	An integrated circuit that integrates all components of a computer
 	or other electronic systems.
 
+_media-glossary-stream:
+    Stream
+	A distinct flow of data (image data or metadata) over a media pipeline
+	from source to sink. A source may be e.g. an image sensor and a sink
+	e.g. a memory buffer.
+
     V4L2 API
 	**V4L2 userspace API**
 
Documentation/userspace-api/media/mediactl/media-types.rst +6 -0
index 0ffeece1e0c8..28941da27790 100644
@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
 .. _MEDIA-PAD-FL-SINK:
 .. _MEDIA-PAD-FL-SOURCE:
 .. _MEDIA-PAD-FL-MUST-CONNECT:
+.. _MEDIA-PAD-FL-INTERNAL:
 
 .. flat-table:: Media pad flags
     :header-rows:  0
@@ -382,6 +383,11 @@ Types and flags used to represent the media graph elements
 	  when this flag isn't set; the absence of the flag doesn't imply
 	  there is none.
 
+    *  -  ``MEDIA_PAD_FL_INTERNAL``
+       -  The internal flag indicates an internal pad that has no external
+	  connections. Such a pad shall not be connected with a link. The
+	  internal flag indicates that the :ref:``stream
+	  <media-glossary-stream>`` either starts or ends in the entity.
 
 One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
 must be set for every pad.
drivers/media/mc/mc-entity.c +2 -1
index 4991281dcccc..03a9e0b8ebab 100644
@@ -1071,7 +1071,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
 
 	for (i = 0; i < entity->num_pads; i++) {
 		if ((entity->pads[i].flags &
-		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
+		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
+		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
 			continue;
 
Quoted 10 lines

drivers/media/mc/mc-entity.c media_create_pad_link(struct media_entity *source, u16 source_pad, · line 1099

Quoted 7 lines
drivers/media/mc/mc-entity.c +2 -0
 		if (entity->pads[i].sig_type == sig_type)
@@ -1094,6 +1095,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
 		return -EINVAL;
 	if (WARN_ON(!(source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE)))
 		return -EINVAL;
+	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE &&
+		    source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL))

Even though this code is correct, I prefer to have () around the 'x & y' parts. It avoids confusion for the reader.

There actually should be a check here that neither of the pads have internal flag set --- internal pads may not be connected by a link. I'll fix that for v4. This was actually a leftover from the "internal source" flag.

Quoted 2 lines

Up to you, though.

I'd prefer to keep it as-is. :-)

Signature · 3 lines
Regards,

Sakari Ailus
#13 lore raw

Re: [PATCH v3 03/10] media: mc: Add INTERNAL pad flag

In reply to Sakari Ailus

Hi Sakari

Quoted 69 lines · Sakari Ailus

Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.

Also prevent creating links to pads that have been flagged as internal.

---
 Documentation/userspace-api/media/glossary.rst             | 6 ++++++
 Documentation/userspace-api/media/mediactl/media-types.rst | 6 ++++++
 drivers/media/mc/mc-entity.c                               | 6 +++++-
 include/uapi/linux/media.h                                 | 1 +
 4 files changed, 18 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/glossary.rst +6 -0
index 96a360edbf3b..f7b99a4527c7 100644
@@ -173,6 +173,12 @@ Glossary
 	An integrated circuit that integrates all components of a computer
 	or other electronic systems.
 
+_media-glossary-stream:
+    Stream
+	A distinct flow of data (image data or metadata) over a media pipeline
+	from source to sink. A source may be e.g. an image sensor and a sink
+	e.g. a memory buffer.
+
     V4L2 API
 	**V4L2 userspace API**
 
Documentation/userspace-api/media/mediactl/media-types.rst +6 -0
index 0ffeece1e0c8..28941da27790 100644
@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
 .. _MEDIA-PAD-FL-SINK:
 .. _MEDIA-PAD-FL-SOURCE:
 .. _MEDIA-PAD-FL-MUST-CONNECT:
+.. _MEDIA-PAD-FL-INTERNAL:
 
 .. flat-table:: Media pad flags
     :header-rows:  0
@@ -382,6 +383,11 @@ Types and flags used to represent the media graph elements
 	  when this flag isn't set; the absence of the flag doesn't imply
 	  there is none.
 
+    *  -  ``MEDIA_PAD_FL_INTERNAL``
+       -  The internal flag indicates an internal pad that has no external
+	  connections. Such a pad shall not be connected with a link. The
+	  internal flag indicates that the :ref:``stream
+	  <media-glossary-stream>`` either starts or ends in the entity.
 
 One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
 must be set for every pad.
drivers/media/mc/mc-entity.c +2 -1
index 4991281dcccc..03a9e0b8ebab 100644
@@ -1071,7 +1071,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
 
 	for (i = 0; i < entity->num_pads; i++) {
 		if ((entity->pads[i].flags &
-		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
+		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
+		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
 			continue;
 
 		if (entity->pads[i].sig_type == sig_type)

drivers/media/mc/mc-entity.c media_create_pad_link(struct media_entity *source, u16 source_pad, · line 1098

Quoted 4 lines
drivers/media/mc/mc-entity.c +1 -0
@@ -1094,6 +1095,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
 		return -EINVAL;
 	if (WARN_ON(!(source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE)))
 		return -EINVAL;
+	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE &&

Why do you need to check for this again here ? Hasn't the previous if() clause just validated that

        source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE ?
Quoted 20 lines
drivers/media/mc/mc-entity.c +2 -0
+		    source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL))
+		return -EINVAL;
 	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
 		return -EINVAL;
 
include/uapi/linux/media.h +1 -1
index 1c80b1d6bbaf..80cfd12a43fc 100644
@@ -208,6 +208,7 @@ struct media_entity_desc {
 #define MEDIA_PAD_FL_SINK			(1U << 0)
 #define MEDIA_PAD_FL_SOURCE			(1U << 1)
 #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
+#define MEDIA_PAD_FL_INTERNAL			(1U << 3)
 
 struct media_pad_desc {
 	__u32 entity;		/* entity ID */
--

2.39.2

author maintainer
#14 lore raw

Re: [PATCH v3 03/10] media: mc: Add INTERNAL pad flag

Hi Jacopo,

Thanks for the review.

Quoted 80 lines · Jacopo Mondi

Hi Sakari

Quoted 69 lines · Sakari Ailus

Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.

Also prevent creating links to pads that have been flagged as internal.

---
 Documentation/userspace-api/media/glossary.rst             | 6 ++++++
 Documentation/userspace-api/media/mediactl/media-types.rst | 6 ++++++
 drivers/media/mc/mc-entity.c                               | 6 +++++-
 include/uapi/linux/media.h                                 | 1 +
 4 files changed, 18 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/glossary.rst +6 -0
index 96a360edbf3b..f7b99a4527c7 100644
@@ -173,6 +173,12 @@ Glossary
 	An integrated circuit that integrates all components of a computer
 	or other electronic systems.
 
+_media-glossary-stream:
+    Stream
+	A distinct flow of data (image data or metadata) over a media pipeline
+	from source to sink. A source may be e.g. an image sensor and a sink
+	e.g. a memory buffer.
+
     V4L2 API
 	**V4L2 userspace API**
 
Documentation/userspace-api/media/mediactl/media-types.rst +6 -0
index 0ffeece1e0c8..28941da27790 100644
@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
 .. _MEDIA-PAD-FL-SINK:
 .. _MEDIA-PAD-FL-SOURCE:
 .. _MEDIA-PAD-FL-MUST-CONNECT:
+.. _MEDIA-PAD-FL-INTERNAL:
 
 .. flat-table:: Media pad flags
     :header-rows:  0
@@ -382,6 +383,11 @@ Types and flags used to represent the media graph elements
 	  when this flag isn't set; the absence of the flag doesn't imply
 	  there is none.
 
+    *  -  ``MEDIA_PAD_FL_INTERNAL``
+       -  The internal flag indicates an internal pad that has no external
+	  connections. Such a pad shall not be connected with a link. The
+	  internal flag indicates that the :ref:``stream
+	  <media-glossary-stream>`` either starts or ends in the entity.
 
 One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
 must be set for every pad.
drivers/media/mc/mc-entity.c +2 -1
index 4991281dcccc..03a9e0b8ebab 100644
@@ -1071,7 +1071,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
 
 	for (i = 0; i < entity->num_pads; i++) {
 		if ((entity->pads[i].flags &
-		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
+		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
+		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
 			continue;
 
 		if (entity->pads[i].sig_type == sig_type)

drivers/media/mc/mc-entity.c media_create_pad_link(struct media_entity *source, u16 source_pad, · line 1098

Quoted 4 lines
drivers/media/mc/mc-entity.c +1 -0
@@ -1094,6 +1095,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
 		return -EINVAL;
 	if (WARN_ON(!(source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE)))
 		return -EINVAL;
+	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE &&

Why do you need to check for this again here ? Hasn't the previous if() clause just validated that

Quoted 1 line
        source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE ?

Yes, this needs to be changed. As a link is created here, the internal flag shouldn't be present in neither sink nor source pads. I may have forgotten to address this previously, I'll do this for v4.

Quoted 18 lines
Quoted 18 lines
drivers/media/mc/mc-entity.c +2 -0
+		    source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL))
+		return -EINVAL;
 	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
 		return -EINVAL;
 
include/uapi/linux/media.h +1 -0
index 1c80b1d6bbaf..80cfd12a43fc 100644
@@ -208,6 +208,7 @@ struct media_entity_desc {
 #define MEDIA_PAD_FL_SINK			(1U << 0)
 #define MEDIA_PAD_FL_SOURCE			(1U << 1)
 #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
+#define MEDIA_PAD_FL_INTERNAL			(1U << 3)
 
 struct media_pad_desc {
 	__u32 entity;		/* entity ID */
Signature · 3 lines
Regards,

Sakari Ailus
maintainer
#15 lore raw

Re: [PATCH v3 03/10] media: mc: Add INTERNAL pad flag

In reply to Sakari Ailus

Hi Sakari,

Thank you for the patch.

Quoted 44 lines · Sakari Ailus

Internal source pads will be used as routing endpoints in V4L2 [GS]_ROUTING IOCTLs, to indicate that the stream begins in the entity.

Also prevent creating links to pads that have been flagged as internal.

---
 Documentation/userspace-api/media/glossary.rst             | 6 ++++++
 Documentation/userspace-api/media/mediactl/media-types.rst | 6 ++++++
 drivers/media/mc/mc-entity.c                               | 6 +++++-
 include/uapi/linux/media.h                                 | 1 +
 4 files changed, 18 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/glossary.rst +6 -0
index 96a360edbf3b..f7b99a4527c7 100644
@@ -173,6 +173,12 @@ Glossary
 	An integrated circuit that integrates all components of a computer
 	or other electronic systems.
 
+_media-glossary-stream:
+    Stream
+	A distinct flow of data (image data or metadata) over a media pipeline
+	from source to sink. A source may be e.g. an image sensor and a sink
+	e.g. a memory buffer.
+
     V4L2 API
 	**V4L2 userspace API**
 
Documentation/userspace-api/media/mediactl/media-types.rst +1 -0
index 0ffeece1e0c8..28941da27790 100644
@@ -361,6 +361,7 @@ Types and flags used to represent the media graph elements
 .. _MEDIA-PAD-FL-SINK:
 .. _MEDIA-PAD-FL-SOURCE:
 .. _MEDIA-PAD-FL-MUST-CONNECT:
+.. _MEDIA-PAD-FL-INTERNAL:
 
 .. flat-table:: Media pad flags
     :header-rows:  0
@@ -382,6 +383,11 @@ Types and flags used to represent the media graph elements
 	  when this flag isn't set; the absence of the flag doesn't imply

Documentation/userspace-api/media/mediactl/media-types.rst Types and flags used to represent the media graph elements · line 393

Quoted 10 lines
Documentation/userspace-api/media/mediactl/media-types.rst +5 -0
 	  there is none.
 
+    *  -  ``MEDIA_PAD_FL_INTERNAL``
+       -  The internal flag indicates an internal pad that has no external
+	  connections. Such a pad shall not be connected with a link. The
+	  internal flag indicates that the :ref:``stream
+	  <media-glossary-stream>`` either starts or ends in the entity.
 
 One and only one of ``MEDIA_PAD_FL_SINK`` and ``MEDIA_PAD_FL_SOURCE``
 must be set for every pad.

You also need to update the definition of "pad" in media-controller-model.rst. That's where you should explain what an internal pad *is*, with enough information to make it understandable to people who haven't worked on this patches series :-) You'll probably need a couple of paragraphs, or possibly adding text below the list of objects.

It would be nice to fix the indentation inconsistency in that file by the way. I'd send a patch, but it will conflict, it would be easier if you could handle it.

Quoted 36 lines
drivers/media/mc/mc-entity.c +5 -1
index 4991281dcccc..03a9e0b8ebab 100644
@@ -1071,7 +1071,8 @@ int media_get_pad_index(struct media_entity *entity, u32 pad_type,
 
 	for (i = 0; i < entity->num_pads; i++) {
 		if ((entity->pads[i].flags &
-		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE)) != pad_type)
+		     (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE |
+		      MEDIA_PAD_FL_INTERNAL)) != pad_type)
 			continue;
 
 		if (entity->pads[i].sig_type == sig_type)
@@ -1094,6 +1095,9 @@ media_create_pad_link(struct media_entity *source, u16 source_pad,
 		return -EINVAL;
 	if (WARN_ON(!(source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE)))
 		return -EINVAL;
+	if (WARN_ON(source->pads[source_pad].flags & MEDIA_PAD_FL_SOURCE &&
+		    source->pads[source_pad].flags & MEDIA_PAD_FL_INTERNAL))
+		return -EINVAL;
 	if (WARN_ON(!(sink->pads[sink_pad].flags & MEDIA_PAD_FL_SINK)))
 		return -EINVAL;
 
include/uapi/linux/media.h +1 -0
index 1c80b1d6bbaf..80cfd12a43fc 100644
@@ -208,6 +208,7 @@ struct media_entity_desc {
 #define MEDIA_PAD_FL_SINK			(1U << 0)
 #define MEDIA_PAD_FL_SOURCE			(1U << 1)
 #define MEDIA_PAD_FL_MUST_CONNECT		(1U << 2)
+#define MEDIA_PAD_FL_INTERNAL			(1U << 3)
 
 struct media_pad_desc {
 	__u32 entity;		/* entity ID */
Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#16 lore raw

PATCH v3 4/10 media: uapi: Add generic serial metadata mbus formats

In reply to Sakari Ailus

Add generic serial metadata mbus formats. These formats describe data width and packing but not the content itself. The reason for specifying such formats is that the formats as such are fairly device specific but they are still handled by CSI-2 receiver drivers that should not be aware of device specific formats. What makes generic metadata formats possible is that these formats are parsed by software only, after capturing the data to system memory.

---
 .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++++++
 include/uapi/linux/media-bus-format.h         |   9 +
 2 files changed, 266 insertions(+)
Documentation/userspace-api/media/v4l/subdev-formats.rst +257 -0
index a3a35eeed708..c615da08502d 100644
@@ -8234,3 +8234,260 @@ The following table lists the existing metadata formats.
 	both sides of the link and the bus format is a fixed
 	metadata format that is not configurable from userspace.
 	Width and height will be set to 0 for this format.
+
+Generic Serial Metadata Formats
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Generic serial metadata formats are used on serial busses where the actual data
+content is more or less device specific but the data is transmitted and received
+by multiple devices that do not process the data in any way, simply writing
+it to system memory for processing in software at the end of the pipeline.
+
+The more specific variant describing the actual data is used on the internal
+source pad of the originating sub-device.
+
+"b" in an array cell signifies a byte of data, followed by the number of byte
+and finally the bit number in subscript. "p" indicates a padding bit.
+
+.. _media-bus-format-generic-meta:
+
+.. cssclass: longtable
+
+.. flat-table:: Generic Serial Metadata Formats
+    :header-rows:  2
+    :stub-columns: 0
+
+    * - Identifier
+      - Code
+      -
+      - :cspan:`23` Data organization
+    * -
+      -
+      - Bit
+      - 23
+      - 22
+      - 21
+      - 20
+      - 19
+      - 18
+      - 17
+      - 16
+      - 15
+      - 14
+      - 13
+      - 12
+      - 11
+      - 10
+      - 9
+      - 8
+      - 7
+      - 6
+      - 5
+      - 4
+      - 3
+      - 2
+      - 1
+      - 0
+    * .. _MEDIA-BUS-FMT-META-8:
+
+      - MEDIA_BUS_FMT_META_8
+      - 0x8001
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+    * .. _MEDIA-BUS-FMT-META-10:
+
+      - MEDIA_BUS_FMT_META_10
+      - 0x8002
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-12:
+
+      - MEDIA_BUS_FMT_META_12
+      - 0x8003
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-14:
+
+      - MEDIA_BUS_FMT_META_14
+      - 0x8004
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-16:
+
+      - MEDIA_BUS_FMT_META_16
+      - 0x8005
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-20:
+
+      - MEDIA_BUS_FMT_META_20
+      - 0x8007
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-24:
+
+      - MEDIA_BUS_FMT_META_24
+      - 0x8009
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
include/uapi/linux/media-bus-format.h +9 -0
index a03c543cb072..9ee031397372 100644
@@ -173,4 +173,13 @@
  */
 #define MEDIA_BUS_FMT_METADATA_FIXED		0x7001
 
+/* Generic line based metadata formats for serial buses. Next is 0x8008. */
+#define MEDIA_BUS_FMT_META_8			0x8001
+#define MEDIA_BUS_FMT_META_10			0x8002
+#define MEDIA_BUS_FMT_META_12			0x8003
+#define MEDIA_BUS_FMT_META_14			0x8004
+#define MEDIA_BUS_FMT_META_16			0x8005
+#define MEDIA_BUS_FMT_META_20			0x8006
+#define MEDIA_BUS_FMT_META_24			0x8007
+
 #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
Signature · 1 line
2.39.2
#17 lore raw

Re: [PATCH v3 04/10] media: uapi: Add generic serial metadata mbus formats

Hi Sakari,

Quoted 7 lines · Sakari Ailus

Add generic serial metadata mbus formats. These formats describe data width and packing but not the content itself. The reason for specifying such formats is that the formats as such are fairly device specific but they are still handled by CSI-2 receiver drivers that should not be aware of device specific formats. What makes generic metadata formats possible is that these formats are parsed by software only, after capturing the data to system memory.

If I'm not mistaken, the CSI-2 spec doesn't say much about embedded data, except that it may exist. Afaics, in CSI-2, the embedded data is split into "lines", although the amount of data can be different than in the video lines.

The CCS specs talks more about embedded data. Some of it is quite odd, like: "The length of the embedded data line shall not exceed the length of the image data line. The embedded data line should have the same length as the image data line.". I think it means the embedded line can be shorter than image line, but not longer.

CCS also says that an embedded line should use 0x07 as padding at the end of the line, if there's less data than the image line.

CCS also talks about how the embedded data would be packed, and in some cases that packing would be the same as for pixel data.

But I don't think these formats are generic. They're defined in CCS, so shouldn't the format be, e.g., MEDIA_BUS_FMT_META_CCS_RAW_12 or such?

The MEDIA_BUS_FMT_META_8 is quite safe one, as it just means byte data without any padding or packing.

Tomi

Quoted 289 lines
---
  .../media/v4l/subdev-formats.rst              | 257 ++++++++++++++++++
  include/uapi/linux/media-bus-format.h         |   9 +
  2 files changed, 266 insertions(+)
Documentation/userspace-api/media/v4l/subdev-formats.rst +257 -0
index a3a35eeed708..c615da08502d 100644
@@ -8234,3 +8234,260 @@ The following table lists the existing metadata formats.
  	both sides of the link and the bus format is a fixed
  	metadata format that is not configurable from userspace.
  	Width and height will be set to 0 for this format.
+
+Generic Serial Metadata Formats
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Generic serial metadata formats are used on serial busses where the actual data
+content is more or less device specific but the data is transmitted and received
+by multiple devices that do not process the data in any way, simply writing
+it to system memory for processing in software at the end of the pipeline.
+
+The more specific variant describing the actual data is used on the internal
+source pad of the originating sub-device.
+
+"b" in an array cell signifies a byte of data, followed by the number of byte
+and finally the bit number in subscript. "p" indicates a padding bit.
+
+.. _media-bus-format-generic-meta:
+
+.. cssclass: longtable
+
+.. flat-table:: Generic Serial Metadata Formats
+    :header-rows:  2
+    :stub-columns: 0
+
+    * - Identifier
+      - Code
+      -
+      - :cspan:`23` Data organization
+    * -
+      -
+      - Bit
+      - 23
+      - 22
+      - 21
+      - 20
+      - 19
+      - 18
+      - 17
+      - 16
+      - 15
+      - 14
+      - 13
+      - 12
+      - 11
+      - 10
+      - 9
+      - 8
+      - 7
+      - 6
+      - 5
+      - 4
+      - 3
+      - 2
+      - 1
+      - 0
+    * .. _MEDIA-BUS-FMT-META-8:
+
+      - MEDIA_BUS_FMT_META_8
+      - 0x8001
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+    * .. _MEDIA-BUS-FMT-META-10:
+
+      - MEDIA_BUS_FMT_META_10
+      - 0x8002
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-12:
+
+      - MEDIA_BUS_FMT_META_12
+      - 0x8003
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-14:
+
+      - MEDIA_BUS_FMT_META_14
+      - 0x8004
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-16:
+
+      - MEDIA_BUS_FMT_META_16
+      - 0x8005
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-20:
+
+      - MEDIA_BUS_FMT_META_20
+      - 0x8007
+      -
+      -
+      -
+      -
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+    * .. _MEDIA-BUS-FMT-META-24:
+
+      - MEDIA_BUS_FMT_META_24
+      - 0x8009
+      -
+      - b0\ :sub:`7`
+      - b0\ :sub:`6`
+      - b0\ :sub:`5`
+      - b0\ :sub:`4`
+      - b0\ :sub:`3`
+      - b0\ :sub:`2`
+      - b0\ :sub:`1`
+      - b0\ :sub:`0`
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
+      - p
include/uapi/linux/media-bus-format.h +9 -0
index a03c543cb072..9ee031397372 100644
@@ -173,4 +173,13 @@
   */
  #define MEDIA_BUS_FMT_METADATA_FIXED		0x7001
 
+/* Generic line based metadata formats for serial buses. Next is 0x8008. */
+#define MEDIA_BUS_FMT_META_8			0x8001
+#define MEDIA_BUS_FMT_META_10			0x8002
+#define MEDIA_BUS_FMT_META_12			0x8003
+#define MEDIA_BUS_FMT_META_14			0x8004
+#define MEDIA_BUS_FMT_META_16			0x8005
+#define MEDIA_BUS_FMT_META_20			0x8006
+#define MEDIA_BUS_FMT_META_24			0x8007
+
  #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
author maintainer
#18 lore raw

Re: [PATCH v3 04/10] media: uapi: Add generic serial metadata mbus formats

Moi,

Quoted 16 lines · Tomi Valkeinen

Hi Sakari,

Quoted 7 lines · Sakari Ailus

Add generic serial metadata mbus formats. These formats describe data width and packing but not the content itself. The reason for specifying such formats is that the formats as such are fairly device specific but they are still handled by CSI-2 receiver drivers that should not be aware of device specific formats. What makes generic metadata formats possible is that these formats are parsed by software only, after capturing the data to system memory.

If I'm not mistaken, the CSI-2 spec doesn't say much about embedded data, except that it may exist. Afaics, in CSI-2, the embedded data is split into "lines", although the amount of data can be different than in the video lines.

Quoted 5 lines

The CCS specs talks more about embedded data. Some of it is quite odd, like:
"The length of the embedded data line shall not exceed the length of the image data line. The embedded data line should have the same length as the image data line.". I think it means the embedded line can be shorter than image line, but not longer.

That's what it means, yes. The CCS also has means to obtain the actual length --- frame format descriptors.

Quoted 6 lines

CCS also says that an embedded line should use 0x07 as padding at the end of the line, if there's less data than the image line.

CCS also talks about how the embedded data would be packed, and in some cases that packing would be the same as for pixel data.

In fact the packing is the same as for pixel data: the CSI-2 does not really differentiate there.

Quoted 3 lines

But I don't think these formats are generic. They're defined in CCS, so shouldn't the format be, e.g., MEDIA_BUS_FMT_META_CCS_RAW_12 or such?

The reason for having generic definitions is that we do not need receiver drivers to be aware of formats that are specific to another driver.

If there is a need for new generic formats that do not match this, we can always add more. But the point is: drivers for devices that do not produce the data should never deal with (device) specific formats.

What comes to CSI-2 and these formats --- on parallel buses you might have the data aligned to the least significant bits instead. But is there a need to transport such data on parallel buses, at least so it would be expressed in mbus formats?

Quoted 3 lines

The MEDIA_BUS_FMT_META_8 is quite safe one, as it just means byte data without any padding or packing.

As you're always dealing with 8 bits only, there is of course less room for variation.

Signature · 3 lines
Regards,

Sakari Ailus
#19 lore raw

Re: [PATCH v3 04/10] media: uapi: Add generic serial metadata mbus formats

Quoted 29 lines · Sakari Ailus

Moi,

Quoted 16 lines · Tomi Valkeinen

Hi Sakari,

Quoted 7 lines · Sakari Ailus

Add generic serial metadata mbus formats. These formats describe data width and packing but not the content itself. The reason for specifying such formats is that the formats as such are fairly device specific but they are still handled by CSI-2 receiver drivers that should not be aware of device specific formats. What makes generic metadata formats possible is that these formats are parsed by software only, after capturing the data to system memory.

If I'm not mistaken, the CSI-2 spec doesn't say much about embedded data, except that it may exist. Afaics, in CSI-2, the embedded data is split into "lines", although the amount of data can be different than in the video lines.

Quoted 5 lines

The CCS specs talks more about embedded data. Some of it is quite odd, like:
"The length of the embedded data line shall not exceed the length of the image data line. The embedded data line should have the same length as the image data line.". I think it means the embedded line can be shorter than image line, but not longer.

That's what it means, yes. The CCS also has means to obtain the actual length --- frame format descriptors.

Quoted 1 line
Quoted 8 lines
Quoted 5 lines

CCS also says that an embedded line should use 0x07 as padding at the end of the line, if there's less data than the image line.

CCS also talks about how the embedded data would be packed, and in some cases that packing would be the same as for pixel data.

In fact the packing is the same as for pixel data: the CSI-2 does not really differentiate there.

If I understand the CCS spec right, the packing is not the same as for the pixel data. You can have RAW12 pixel data but 8-byte embedded data. But if you meant that the different packing style options are the same for pixel and embedded data, yes.

Quoted 5 lines
Quoted 2 lines

But I don't think these formats are generic. They're defined in CCS, so shouldn't the format be, e.g., MEDIA_BUS_FMT_META_CCS_RAW_12 or such?

The reason for having generic definitions is that we do not need receiver drivers to be aware of formats that are specific to another driver.

Yes, I agree with that, and that's not my point here. But "generic" doesn't mean the definitions are not for some particular bus or standard, "generic" just means that it doesn't specify the content, only the packing.

My point is that these packings seem to be specific to CCS. While non-CCS compliant sensors may use the same packing, the packing itself is still a "CCS packing". So why not call them that? The 8-bit format is fully generic, whereas the rest are CCS packings, as defined in the CCS spec (the CCS spec also specifies the content, but here were only using the CCS packing).

Maybe they shouldn't be called "generic", but... umm... Content unaware metadata formats... doesn't sound very good =).

Also, I'm a bit confused about CSI-2 pixel and embedded data formats and how we handle them.

For MEDIA_BUS_FMT_SBGGR10_1X10, we define that the data contains 10 bits per pixel, from bit 0 to 9. But CSI-2, for RAW10, actually sends it differently, with the higher bits first, and the lowest bits in the fourth byte. So that CSI-2 packing is implicit, kind of hidden here. Which is probably fine, as we're really only interested in the unpacked data, not the CSI-2 packed. And when writing this data to memory, the DMA engine can write it either as is, or unpack each pixel to a 16-bit container.

For MEDIA_BUS_FMT_META_10, we define it similarly to MEDIA_BUS_FMT_SBGGR10_1X10, except the lowest 2 bits are marked as padding bits. And, I think, MEDIA_BUS_FMT_META_10 implies RAW10 CSI-2 format. However, when writing the data to memory, we don't want either of the modes used in the above pixel data case, but rather we want to write the data as it is in the CSI-2 bus. So, the DMA engine can either reverse the RAW10 unpacking to get the wanted output format, or alternatively the CSI-2 receiver could instead use RAW8 mode to avoid any unpacking.

Does the above make any sense? I'm a bit confused about all the packings and unpackings =). Does MEDIA_BUS_FMT_META_10 mean that the CSI-2 TX uses RAW10 CSI-2 packing, but the receiver uses RAW8 CSI-2 packing?

Quoted 8 lines

If there is a need for new generic formats that do not match this, we can always add more. But the point is: drivers for devices that do not produce the data should never deal with (device) specific formats.

What comes to CSI-2 and these formats --- on parallel buses you might have the data aligned to the least significant bits instead. But is there a need to transport such data on parallel buses, at least so it would be expressed in mbus formats?

I have a parallel sensor that sends embedded data. I'll have a look how it organizes the data.

Tomi

maintainer
#20 lore raw

Re: [PATCH v3 04/10] media: uapi: Add generic serial metadata mbus formats

Quoted 7 lines · Tomi Valkeinen
Quoted 6 lines · Sakari Ailus
Quoted 5 lines · Tomi Valkeinen
Quoted 4 lines · Sakari Ailus

Add generic serial metadata mbus formats. These formats describe data width and packing but not the content itself. The reason for specifying such formats is that the formats as such are fairly device specific but they are still handled by CSI-2 receiver drivers that should not be aware

Do we want these formats to be CSI-2-specific ?

Quoted 24 lines
Quoted 24 lines
Quoted 9 lines
Quoted 3 lines

of device specific formats. What makes generic metadata formats possible is that these formats are parsed by software only, after capturing the data to system memory.

If I'm not mistaken, the CSI-2 spec doesn't say much about embedded data, except that it may exist. Afaics, in CSI-2, the embedded data is split into "lines", although the amount of data can be different than in the video lines.

Quoted 5 lines

The CCS specs talks more about embedded data. Some of it is quite odd, like:
"The length of the embedded data line shall not exceed the length of the image data line. The embedded data line should have the same length as the image data line.". I think it means the embedded line can be shorter than image line, but not longer.

That's what it means, yes. The CCS also has means to obtain the actual length --- frame format descriptors.

Quoted 5 lines

CCS also says that an embedded line should use 0x07 as padding at the end of the line, if there's less data than the image line.

CCS also talks about how the embedded data would be packed, and in some cases that packing would be the same as for pixel data.

Quoted 7 lines
Quoted 2 lines

In fact the packing is the same as for pixel data: the CSI-2 does not really differentiate there.

If I understand the CCS spec right, the packing is not the same as for the pixel data. You can have RAW12 pixel data but 8-byte embedded data. But if you meant that the different packing style options are the same for pixel and embedded data, yes.

The idea is to allow CSI-2 receivers to unpack embedded data the same way as pixel data at the hardware level, and guarantee that no embedded data significant byte gets split across multiple samples. When unpacking RAW10 pixel data, the CSI-2 receiver will take 5 bytes and output 4 samples on a 10-bit wide bus. For embedded data, it can conceptually do that same, and output embedded data with one byte per sample, aligned on the MSB of the 10-bit bus. However, in practice, I think the CSI-2 receiver will simply receive the bytes and send them to the DMA engine, without unpacking and then repacking.

This can also be viewed as away to allow CSI-2 transmitters to serialize pixel data and embedded data the same way. For a RAW10 sensor, for instance, the embedded data would be generated on an internal 10-bit wide bus, aligned to the MSB, and would go through CSI-2 packing the same way that pixels do. That's nice, but it's internal to the sensor, so we don't need to expose it to userspace on that side.

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

But I don't think these formats are generic. They're defined in CCS, so shouldn't the format be, e.g., MEDIA_BUS_FMT_META_CCS_RAW_12 or such?

The reason for having generic definitions is that we do not need receiver drivers to be aware of formats that are specific to another driver.

Quoted 8 lines

Yes, I agree with that, and that's not my point here. But "generic" doesn't mean the definitions are not for some particular bus or standard, "generic" just means that it doesn't specify the content, only the packing.

My point is that these packings seem to be specific to CCS. While non-CCS compliant sensors may use the same packing, the packing itself is still a "CCS packing".

Agreed.

Quoted 4 lines

So why not call them that? The 8-bit format is
fully generic, whereas the rest are CCS packings, as defined in the CCS spec (the CCS spec also specifies the content, but here were only using the CCS packing).

I think an additional question is whether we actually need multiple media bus codes here. Are there CSI-2 receivers, or other downstream blocks, that will need to be configured differently to receive embedded data from a CCS (in the sense of using CCS packing, not being fully CCS-compliant) RAW10 sensor compared to a RAW12 sensor ? Or will the CSI-2 receiver just send bytes to memory without caring ? In the latter case, it will be the application that will need to know how the data is packed to ignore one byte out of three for RAW12 and out of five for RAW10.

Quoted 10 lines

Maybe they shouldn't be called "generic", but... umm... Content unaware metadata formats... doesn't sound very good =).

Also, I'm a bit confused about CSI-2 pixel and embedded data formats and how we handle them.

For MEDIA_BUS_FMT_SBGGR10_1X10, we define that the data contains 10 bits per pixel, from bit 0 to 9. But CSI-2, for RAW10, actually sends it differently, with the higher bits first, and the lowest bits in the fourth byte. So that CSI-2 packing is implicit, kind of hidden here.

Usage of MEDIA_BUS_FMT_SBGGR10_1X10 for CSI-2 is a historical decision (or mistake, depending on how you look at it). It doesn't match what actually happens on the bus.

Quoted 9 lines

Which is probably fine, as we're really only interested in the unpacked data, not the CSI-2 packed. And when writing this data to memory, the DMA engine can write it either as is, or unpack each pixel to a 16-bit container.

For MEDIA_BUS_FMT_META_10, we define it similarly to MEDIA_BUS_FMT_SBGGR10_1X10, except the lowest 2 bits are marked as padding bits. And, I think, MEDIA_BUS_FMT_META_10 implies RAW10 CSI-2 format.

That's how I understand it too

Quoted 6 lines

However, when writing the data to memory, we don't want either of the modes used in the above pixel data case, but rather we want to write the data as it is in the CSI-2 bus. So, the DMA engine can either reverse the RAW10 unpacking to get the wanted output format, or alternatively the CSI-2 receiver could instead use RAW8 mode to avoid any unpacking.

I don't think the DMA engine would "reverse the RAW10 packing" here. It will write bytes as they arrive (or more likely, a set of bytes at a time to operate on bursts). The other option would be for the DMA engine to drop the padding bytes, but I don't know of any CSI-2 receiver that can do that.

Quoted 15 lines

Does the above make any sense? I'm a bit confused about all the packings and unpackings =). Does MEDIA_BUS_FMT_META_10 mean that the CSI-2 TX uses RAW10 CSI-2 packing, but the receiver uses RAW8 CSI-2 packing?

Quoted 8 lines

If there is a need for new generic formats that do not match this, we can always add more. But the point is: drivers for devices that do not produce the data should never deal with (device) specific formats.

What comes to CSI-2 and these formats --- on parallel buses you might have the data aligned to the least significant bits instead. But is there a need to transport such data on parallel buses, at least so it would be expressed in mbus formats?

I have a parallel sensor that sends embedded data. I'll have a look how it organizes the data.

Signature · 3 lines
Regards,

Laurent Pinchart
#21 lore raw

Re: [PATCH v3 04/10] media: uapi: Add generic serial metadata mbus formats

Hi,

Quoted 38 lines · Laurent Pinchart
Quoted 7 lines · Tomi Valkeinen
Quoted 6 lines · Sakari Ailus
Quoted 5 lines · Tomi Valkeinen
Quoted 4 lines · Sakari Ailus

Add generic serial metadata mbus formats. These formats describe data width and packing but not the content itself. The reason for specifying such formats is that the formats as such are fairly device specific but they are still handled by CSI-2 receiver drivers that should not be aware

Do we want these formats to be CSI-2-specific ?

Quoted 27 lines
Quoted 26 lines
Quoted 9 lines
Quoted 3 lines

of device specific formats. What makes generic metadata formats possible is that these formats are parsed by software only, after capturing the data to system memory.

If I'm not mistaken, the CSI-2 spec doesn't say much about embedded data, except that it may exist. Afaics, in CSI-2, the embedded data is split into "lines", although the amount of data can be different than in the video lines.

Quoted 5 lines

The CCS specs talks more about embedded data. Some of it is quite odd, like:
"The length of the embedded data line shall not exceed the length of the image data line. The embedded data line should have the same length as the image data line.". I think it means the embedded line can be shorter than image line, but not longer.

That's what it means, yes. The CCS also has means to obtain the actual length --- frame format descriptors.

Quoted 5 lines

CCS also says that an embedded line should use 0x07 as padding at the end of the line, if there's less data than the image line.

CCS also talks about how the embedded data would be packed, and in some cases that packing would be the same as for pixel data.

In fact the packing is the same as for pixel data: the CSI-2 does not really differentiate there.

Quoted 7 lines
Quoted 4 lines

If I understand the CCS spec right, the packing is not the same as for the pixel data. You can have RAW12 pixel data but 8-byte embedded data. But if you meant that the different packing style options are the same for pixel and embedded data, yes.

The idea is to allow CSI-2 receivers to unpack embedded data the same way as pixel data at the hardware level, and guarantee that no embedded

Yes, but my comment was to Sakari's "the packing is the same as for pixel data" comment. Afaics, there's nothing in CSI-2 or CCS that says the packing is the same for pixel data and embedded data.

In fact, the CSS says "By default, top-embedded data shall use the same data packing as the Visible pixel data with one embedded data byte per pixel", implying that it would not always be the case. It continues that for RAW16/RAW20/RAW24 pixels the embedded data could be sent with RAW8/RAW10/RAW12.

That was just for top-embedded data, and I couldn't right away see mention of other embedded data.

Quoted 7 lines

data significant byte gets split across multiple samples. When unpacking RAW10 pixel data, the CSI-2 receiver will take 5 bytes and output 4 samples on a 10-bit wide bus. For embedded data, it can conceptually do that same, and output embedded data with one byte per sample, aligned on the MSB of the 10-bit bus. However, in practice, I think the CSI-2 receiver will simply receive the bytes and send them to the DMA engine, without unpacking and then repacking.

At least TI CAL CSI-2 RX can extract the data from the CSI-2 stream using different unpacking for pixel and embedded data, and then (optionally) pack it in different ways when writing to memory.

Quoted 29 lines

This can also be viewed as away to allow CSI-2 transmitters to serialize pixel data and embedded data the same way. For a RAW10 sensor, for instance, the embedded data would be generated on an internal 10-bit wide bus, aligned to the MSB, and would go through CSI-2 packing the same way that pixels do. That's nice, but it's internal to the sensor, so we don't need to expose it to userspace on that side.

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

But I don't think these formats are generic. They're defined in CCS, so shouldn't the format be, e.g., MEDIA_BUS_FMT_META_CCS_RAW_12 or such?

The reason for having generic definitions is that we do not need receiver drivers to be aware of formats that are specific to another driver.

Quoted 8 lines

Yes, I agree with that, and that's not my point here. But "generic" doesn't mean the definitions are not for some particular bus or standard, "generic" just means that it doesn't specify the content, only the packing.

My point is that these packings seem to be specific to CCS. While non-CCS compliant sensors may use the same packing, the packing itself is still a "CCS packing".

Agreed.

Quoted 4 lines

So why not call them that? The 8-bit format is
fully generic, whereas the rest are CCS packings, as defined in the CCS spec (the CCS spec also specifies the content, but here were only using the CCS packing).

Quoted 9 lines

I think an additional question is whether we actually need multiple media bus codes here. Are there CSI-2 receivers, or other downstream blocks, that will need to be configured differently to receive embedded data from a CCS (in the sense of using CCS packing, not being fully CCS-compliant) RAW10 sensor compared to a RAW12 sensor ? Or will the CSI-2 receiver just send bytes to memory without caring ? In the latter case, it will be the application that will need to know how the data is packed to ignore one byte out of three for RAW12 and out of five for RAW10.

I would presume all receivers can operate in "just write what you receive" mode, in which case the packing doesn't matter as you say, but then again, I think it's safer to have mbus codes for the different packings.

Quoted 34 lines
Quoted 10 lines

Maybe they shouldn't be called "generic", but... umm... Content unaware metadata formats... doesn't sound very good =).

Also, I'm a bit confused about CSI-2 pixel and embedded data formats and how we handle them.

For MEDIA_BUS_FMT_SBGGR10_1X10, we define that the data contains 10 bits per pixel, from bit 0 to 9. But CSI-2, for RAW10, actually sends it differently, with the higher bits first, and the lowest bits in the fourth byte. So that CSI-2 packing is implicit, kind of hidden here.

Usage of MEDIA_BUS_FMT_SBGGR10_1X10 for CSI-2 is a historical decision (or mistake, depending on how you look at it). It doesn't match what actually happens on the bus.

Quoted 9 lines

Which is probably fine, as we're really only interested in the unpacked data, not the CSI-2 packed. And when writing this data to memory, the DMA engine can write it either as is, or unpack each pixel to a 16-bit container.

For MEDIA_BUS_FMT_META_10, we define it similarly to MEDIA_BUS_FMT_SBGGR10_1X10, except the lowest 2 bits are marked as padding bits. And, I think, MEDIA_BUS_FMT_META_10 implies RAW10 CSI-2 format.

That's how I understand it too

Quoted 6 lines

However, when writing the data to memory, we don't want either of the modes used in the above pixel data case, but rather we want to write the data as it is in the CSI-2 bus. So, the DMA engine can either reverse the RAW10 unpacking to get the wanted output format, or alternatively the CSI-2 receiver could instead use RAW8 mode to avoid any unpacking.

Quoted 5 lines

I don't think the DMA engine would "reverse the RAW10 packing" here. It will write bytes as they arrive (or more likely, a set of bytes at a time to operate on bursts). The other option would be for the DMA engine to drop the padding bytes, but I don't know of any CSI-2 receiver that can do that.

Well, for pixel data, TI CAL has the following options when writing to memory for, e.g. RAW12:
- RAW12 MIPI (a separate byte for lsbs)
- RAW12 Linear (12 consecutive bits per pixel)
- RAW16 (12 consecutive bits per pixel in a 16bit container)

Of these, only the RAW12 MIPI makes sense for embedded data (actually the whole processing block can be and should be skipped for embedded data, so it's more like CAL is receiving RAW8 and writing out RAW8).

Or, if it would be possible, skipping the padding bytes. Which looks like it is almost possible with TI CAL, but not quite: it can write with RAW8, but afaics that always takes the lowest bits of the 12-bit pixel, but here we'd like it to take the highest bits.

Anyway, I don't think there's any issue here, I was just writing down my thoughts. As long as we have good mbus formats which describe the packing fully, and V4L2 formats which describe the memory formats, we should be fine.

Tomi

Quoted 16 lines
Quoted 15 lines

Does the above make any sense? I'm a bit confused about all the packings and unpackings =). Does MEDIA_BUS_FMT_META_10 mean that the CSI-2 TX uses RAW10 CSI-2 packing, but the receiver uses RAW8 CSI-2 packing?

Quoted 8 lines

If there is a need for new generic formats that do not match this, we can always add more. But the point is: drivers for devices that do not produce the data should never deal with (device) specific formats.

What comes to CSI-2 and these formats --- on parallel buses you might have the data aligned to the least significant bits instead. But is there a need to transport such data on parallel buses, at least so it would be expressed in mbus formats?

I have a parallel sensor that sends embedded data. I'll have a look how it organizes the data.

maintainer
#22 lore raw

Re: [PATCH v3 04/10] media: uapi: Add generic serial metadata mbus formats

Hi Tomi,

Quoted 51 lines · Tomi Valkeinen
Quoted 38 lines · Laurent Pinchart
Quoted 7 lines · Tomi Valkeinen
Quoted 6 lines · Sakari Ailus
Quoted 5 lines · Tomi Valkeinen
Quoted 4 lines · Sakari Ailus

Add generic serial metadata mbus formats. These formats describe data width and packing but not the content itself. The reason for specifying such formats is that the formats as such are fairly device specific but they are still handled by CSI-2 receiver drivers that should not be aware

Do we want these formats to be CSI-2-specific ?

Quoted 27 lines
Quoted 26 lines
Quoted 9 lines
Quoted 3 lines

of device specific formats. What makes generic metadata formats possible is that these formats are parsed by software only, after capturing the data to system memory.

If I'm not mistaken, the CSI-2 spec doesn't say much about embedded data, except that it may exist. Afaics, in CSI-2, the embedded data is split into "lines", although the amount of data can be different than in the video lines.

Quoted 5 lines

The CCS specs talks more about embedded data. Some of it is quite odd, like:
"The length of the embedded data line shall not exceed the length of the image data line. The embedded data line should have the same length as the image data line.". I think it means the embedded line can be shorter than image line, but not longer.

That's what it means, yes. The CCS also has means to obtain the actual length --- frame format descriptors.

Quoted 5 lines

CCS also says that an embedded line should use 0x07 as padding at the end of the line, if there's less data than the image line.

CCS also talks about how the embedded data would be packed, and in some cases that packing would be the same as for pixel data.

In fact the packing is the same as for pixel data: the CSI-2 does not really differentiate there.

Quoted 7 lines
Quoted 4 lines

If I understand the CCS spec right, the packing is not the same as for the pixel data. You can have RAW12 pixel data but 8-byte embedded data. But if you meant that the different packing style options are the same for pixel and embedded data, yes.

The idea is to allow CSI-2 receivers to unpack embedded data the same way as pixel data at the hardware level, and guarantee that no embedded

Yes, but my comment was to Sakari's "the packing is the same as for pixel data" comment. Afaics, there's nothing in CSI-2 or CCS that says the packing is the same for pixel data and embedded data.

Quoted 5 lines

In fact, the CSS says "By default, top-embedded data shall use the same data packing as the Visible pixel data with one embedded data byte per pixel", implying that it would not always be the case. It continues that for RAW16/RAW20/RAW24 pixels the embedded data could be sent with RAW8/RAW10/RAW12.

The spec if a bit ambiguous :-S "by default" implies that something else could be done, but it's not clear what the other options are.

Quoted 2 lines

That was just for top-embedded data, and I couldn't right away see mention of other embedded data.

The specification also mentions PDAF data for instance.

Quoted 11 lines
Quoted 7 lines

data significant byte gets split across multiple samples. When unpacking RAW10 pixel data, the CSI-2 receiver will take 5 bytes and output 4 samples on a 10-bit wide bus. For embedded data, it can conceptually do that same, and output embedded data with one byte per sample, aligned on the MSB of the 10-bit bus. However, in practice, I think the CSI-2 receiver will simply receive the bytes and send them to the DMA engine, without unpacking and then repacking.

At least TI CAL CSI-2 RX can extract the data from the CSI-2 stream using different unpacking for pixel and embedded data, and then (optionally) pack it in different ways when writing to memory.

Does this mean that the CAL could take RAW10 embedded data and drop the padding bytes when writing to memory ?

Quoted 90 lines
Quoted 29 lines

This can also be viewed as away to allow CSI-2 transmitters to serialize pixel data and embedded data the same way. For a RAW10 sensor, for instance, the embedded data would be generated on an internal 10-bit wide bus, aligned to the MSB, and would go through CSI-2 packing the same way that pixels do. That's nice, but it's internal to the sensor, so we don't need to expose it to userspace on that side.

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

But I don't think these formats are generic. They're defined in CCS, so shouldn't the format be, e.g., MEDIA_BUS_FMT_META_CCS_RAW_12 or such?

The reason for having generic definitions is that we do not need receiver drivers to be aware of formats that are specific to another driver.

Quoted 8 lines

Yes, I agree with that, and that's not my point here. But "generic" doesn't mean the definitions are not for some particular bus or standard, "generic" just means that it doesn't specify the content, only the packing.

My point is that these packings seem to be specific to CCS. While non-CCS compliant sensors may use the same packing, the packing itself is still a "CCS packing".

Agreed.

Quoted 4 lines

So why not call them that? The 8-bit format is
fully generic, whereas the rest are CCS packings, as defined in the CCS spec (the CCS spec also specifies the content, but here were only using the CCS packing).

Quoted 9 lines

I think an additional question is whether we actually need multiple media bus codes here. Are there CSI-2 receivers, or other downstream blocks, that will need to be configured differently to receive embedded data from a CCS (in the sense of using CCS packing, not being fully CCS-compliant) RAW10 sensor compared to a RAW12 sensor ? Or will the CSI-2 receiver just send bytes to memory without caring ? In the latter case, it will be the application that will need to know how the data is packed to ignore one byte out of three for RAW12 and out of five for RAW10.

I would presume all receivers can operate in "just write what you receive" mode, in which case the packing doesn't matter as you say, but then again, I think it's safer to have mbus codes for the different packings.

Quoted 34 lines
Quoted 10 lines

Maybe they shouldn't be called "generic", but... umm... Content unaware metadata formats... doesn't sound very good =).

Also, I'm a bit confused about CSI-2 pixel and embedded data formats and how we handle them.

For MEDIA_BUS_FMT_SBGGR10_1X10, we define that the data contains 10 bits per pixel, from bit 0 to 9. But CSI-2, for RAW10, actually sends it differently, with the higher bits first, and the lowest bits in the fourth byte. So that CSI-2 packing is implicit, kind of hidden here.

Usage of MEDIA_BUS_FMT_SBGGR10_1X10 for CSI-2 is a historical decision (or mistake, depending on how you look at it). It doesn't match what actually happens on the bus.

Quoted 9 lines

Which is probably fine, as we're really only interested in the unpacked data, not the CSI-2 packed. And when writing this data to memory, the DMA engine can write it either as is, or unpack each pixel to a 16-bit container.

For MEDIA_BUS_FMT_META_10, we define it similarly to MEDIA_BUS_FMT_SBGGR10_1X10, except the lowest 2 bits are marked as padding bits. And, I think, MEDIA_BUS_FMT_META_10 implies RAW10 CSI-2 format.

That's how I understand it too

Quoted 6 lines

However, when writing the data to memory, we don't want either of the modes used in the above pixel data case, but rather we want to write the data as it is in the CSI-2 bus. So, the DMA engine can either reverse the RAW10 unpacking to get the wanted output format, or alternatively the CSI-2 receiver could instead use RAW8 mode to avoid any unpacking.

Quoted 5 lines

I don't think the DMA engine would "reverse the RAW10 packing" here. It will write bytes as they arrive (or more likely, a set of bytes at a time to operate on bursts). The other option would be for the DMA engine to drop the padding bytes, but I don't know of any CSI-2 receiver that can do that.

Well, for pixel data, TI CAL has the following options when writing to memory for, e.g. RAW12:
- RAW12 MIPI (a separate byte for lsbs)
- RAW12 Linear (12 consecutive bits per pixel)
- RAW16 (12 consecutive bits per pixel in a 16bit container)

Quoted 8 lines

Of these, only the RAW12 MIPI makes sense for embedded data (actually the whole processing block can be and should be skipped for embedded data, so it's more like CAL is receiving RAW8 and writing out RAW8).

Or, if it would be possible, skipping the padding bytes. Which looks like it is almost possible with TI CAL, but not quite: it can write with RAW8, but afaics that always takes the lowest bits of the 12-bit pixel, but here we'd like it to take the highest bits.

That's a weird one, even for image data, dropping the MSBs doesn't make much sense, while writing the RAW8 MSB when receiving RAW10 or RAW12 would be useful.

Quoted 20 lines

Anyway, I don't think there's any issue here, I was just writing down my thoughts. As long as we have good mbus formats which describe the packing fully, and V4L2 formats which describe the memory formats, we should be fine.

Quoted 15 lines
Quoted 15 lines

Does the above make any sense? I'm a bit confused about all the packings and unpackings =). Does MEDIA_BUS_FMT_META_10 mean that the CSI-2 TX uses RAW10 CSI-2 packing, but the receiver uses RAW8 CSI-2 packing?

Quoted 8 lines

If there is a need for new generic formats that do not match this, we can always add more. But the point is: drivers for devices that do not produce the data should never deal with (device) specific formats.

What comes to CSI-2 and these formats --- on parallel buses you might have the data aligned to the least significant bits instead. But is there a need to transport such data on parallel buses, at least so it would be expressed in mbus formats?

I have a parallel sensor that sends embedded data. I'll have a look how it organizes the data.

Signature · 3 lines
Regards,

Laurent Pinchart
#23 lore raw

Re: [PATCH v3 04/10] media: uapi: Add generic serial metadata mbus formats

Quoted 177 lines · Laurent Pinchart

Hi Tomi,

Quoted 51 lines · Tomi Valkeinen
Quoted 38 lines · Laurent Pinchart
Quoted 7 lines · Tomi Valkeinen
Quoted 6 lines · Sakari Ailus
Quoted 5 lines · Tomi Valkeinen
Quoted 4 lines · Sakari Ailus

Add generic serial metadata mbus formats. These formats describe data width and packing but not the content itself. The reason for specifying such formats is that the formats as such are fairly device specific but they are still handled by CSI-2 receiver drivers that should not be aware

Do we want these formats to be CSI-2-specific ?

Quoted 27 lines
Quoted 26 lines
Quoted 9 lines
Quoted 3 lines

of device specific formats. What makes generic metadata formats possible is that these formats are parsed by software only, after capturing the data to system memory.

If I'm not mistaken, the CSI-2 spec doesn't say much about embedded data, except that it may exist. Afaics, in CSI-2, the embedded data is split into "lines", although the amount of data can be different than in the video lines.

Quoted 5 lines

The CCS specs talks more about embedded data. Some of it is quite odd, like:
"The length of the embedded data line shall not exceed the length of the image data line. The embedded data line should have the same length as the image data line.". I think it means the embedded line can be shorter than image line, but not longer.

That's what it means, yes. The CCS also has means to obtain the actual length --- frame format descriptors.

Quoted 5 lines

CCS also says that an embedded line should use 0x07 as padding at the end of the line, if there's less data than the image line.

CCS also talks about how the embedded data would be packed, and in some cases that packing would be the same as for pixel data.

In fact the packing is the same as for pixel data: the CSI-2 does not really differentiate there.

Quoted 7 lines
Quoted 4 lines

If I understand the CCS spec right, the packing is not the same as for the pixel data. You can have RAW12 pixel data but 8-byte embedded data. But if you meant that the different packing style options are the same for pixel and embedded data, yes.

The idea is to allow CSI-2 receivers to unpack embedded data the same way as pixel data at the hardware level, and guarantee that no embedded

Yes, but my comment was to Sakari's "the packing is the same as for pixel data" comment. Afaics, there's nothing in CSI-2 or CCS that says the packing is the same for pixel data and embedded data.

Quoted 5 lines

In fact, the CSS says "By default, top-embedded data shall use the same data packing as the Visible pixel data with one embedded data byte per pixel", implying that it would not always be the case. It continues that for RAW16/RAW20/RAW24 pixels the embedded data could be sent with RAW8/RAW10/RAW12.

The spec if a bit ambiguous :-S "by default" implies that something else could be done, but it's not clear what the other options are.

Quoted 2 lines

That was just for top-embedded data, and I couldn't right away see mention of other embedded data.

The specification also mentions PDAF data for instance.

Quoted 11 lines
Quoted 7 lines

data significant byte gets split across multiple samples. When unpacking RAW10 pixel data, the CSI-2 receiver will take 5 bytes and output 4 samples on a 10-bit wide bus. For embedded data, it can conceptually do that same, and output embedded data with one byte per sample, aligned on the MSB of the 10-bit bus. However, in practice, I think the CSI-2 receiver will simply receive the bytes and send them to the DMA engine, without unpacking and then repacking.

At least TI CAL CSI-2 RX can extract the data from the CSI-2 stream using different unpacking for pixel and embedded data, and then (optionally) pack it in different ways when writing to memory.

Does this mean that the CAL could take RAW10 embedded data and drop the padding bytes when writing to memory ?

Quoted 94 lines
Quoted 29 lines

This can also be viewed as away to allow CSI-2 transmitters to serialize pixel data and embedded data the same way. For a RAW10 sensor, for instance, the embedded data would be generated on an internal 10-bit wide bus, aligned to the MSB, and would go through CSI-2 packing the same way that pixels do. That's nice, but it's internal to the sensor, so we don't need to expose it to userspace on that side.

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

But I don't think these formats are generic. They're defined in CCS, so shouldn't the format be, e.g., MEDIA_BUS_FMT_META_CCS_RAW_12 or such?

The reason for having generic definitions is that we do not need receiver drivers to be aware of formats that are specific to another driver.

Quoted 8 lines

Yes, I agree with that, and that's not my point here. But "generic" doesn't mean the definitions are not for some particular bus or standard, "generic" just means that it doesn't specify the content, only the packing.

My point is that these packings seem to be specific to CCS. While non-CCS compliant sensors may use the same packing, the packing itself is still a "CCS packing".

Agreed.

Quoted 4 lines

So why not call them that? The 8-bit format is
fully generic, whereas the rest are CCS packings, as defined in the CCS spec (the CCS spec also specifies the content, but here were only using the CCS packing).

Quoted 9 lines

I think an additional question is whether we actually need multiple media bus codes here. Are there CSI-2 receivers, or other downstream blocks, that will need to be configured differently to receive embedded data from a CCS (in the sense of using CCS packing, not being fully CCS-compliant) RAW10 sensor compared to a RAW12 sensor ? Or will the CSI-2 receiver just send bytes to memory without caring ? In the latter case, it will be the application that will need to know how the data is packed to ignore one byte out of three for RAW12 and out of five for RAW10.

I would presume all receivers can operate in "just write what you receive" mode, in which case the packing doesn't matter as you say, but then again, I think it's safer to have mbus codes for the different packings.

Quoted 34 lines
Quoted 10 lines

Maybe they shouldn't be called "generic", but... umm... Content unaware metadata formats... doesn't sound very good =).

Also, I'm a bit confused about CSI-2 pixel and embedded data formats and how we handle them.

For MEDIA_BUS_FMT_SBGGR10_1X10, we define that the data contains 10 bits per pixel, from bit 0 to 9. But CSI-2, for RAW10, actually sends it differently, with the higher bits first, and the lowest bits in the fourth byte. So that CSI-2 packing is implicit, kind of hidden here.

Usage of MEDIA_BUS_FMT_SBGGR10_1X10 for CSI-2 is a historical decision (or mistake, depending on how you look at it). It doesn't match what actually happens on the bus.

Quoted 9 lines

Which is probably fine, as we're really only interested in the unpacked data, not the CSI-2 packed. And when writing this data to memory, the DMA engine can write it either as is, or unpack each pixel to a 16-bit container.

For MEDIA_BUS_FMT_META_10, we define it similarly to MEDIA_BUS_FMT_SBGGR10_1X10, except the lowest 2 bits are marked as padding bits. And, I think, MEDIA_BUS_FMT_META_10 implies RAW10 CSI-2 format.

That's how I understand it too

Quoted 6 lines

However, when writing the data to memory, we don't want either of the modes used in the above pixel data case, but rather we want to write the data as it is in the CSI-2 bus. So, the DMA engine can either reverse the RAW10 unpacking to get the wanted output format, or alternatively the CSI-2 receiver could instead use RAW8 mode to avoid any unpacking.

Quoted 5 lines

I don't think the DMA engine would "reverse the RAW10 packing" here. It will write bytes as they arrive (or more likely, a set of bytes at a time to operate on bursts). The other option would be for the DMA engine to drop the padding bytes, but I don't know of any CSI-2 receiver that can do that.

Well, for pixel data, TI CAL has the following options when writing to memory for, e.g. RAW12:
- RAW12 MIPI (a separate byte for lsbs)
- RAW12 Linear (12 consecutive bits per pixel)
- RAW16 (12 consecutive bits per pixel in a 16bit container)

Of these, only the RAW12 MIPI makes sense for embedded data (actually the whole processing block can be and should be skipped for embedded data, so it's more like CAL is receiving RAW8 and writing out RAW8).

Quoted 8 lines
Quoted 4 lines

Or, if it would be possible, skipping the padding bytes. Which looks like it is almost possible with TI CAL, but not quite: it can write with RAW8, but afaics that always takes the lowest bits of the 12-bit pixel, but here we'd like it to take the highest bits.

That's a weird one, even for image data, dropping the MSBs doesn't make much sense, while writing the RAW8 MSB when receiving RAW10 or RAW12 would be useful.

I think the RAW8 output is designed to be used only for RAW8 input. I haven't confirmed this behavior, but the doc says that RAW8 output writes bits 0-7. And if the input is RAW10/12, then that would imply that the MSBs are dropped.

I could try this at some point, as it's not quite clear from the docs.

Tomi

Quoted 21 lines
Quoted 20 lines

Anyway, I don't think there's any issue here, I was just writing down my thoughts. As long as we have good mbus formats which describe the packing fully, and V4L2 formats which describe the memory formats, we should be fine.

Quoted 15 lines
Quoted 15 lines

Does the above make any sense? I'm a bit confused about all the packings and unpackings =). Does MEDIA_BUS_FMT_META_10 mean that the CSI-2 TX uses RAW10 CSI-2 packing, but the receiver uses RAW8 CSI-2 packing?

Quoted 8 lines

If there is a need for new generic formats that do not match this, we can always add more. But the point is: drivers for devices that do not produce the data should never deal with (device) specific formats.

What comes to CSI-2 and these formats --- on parallel buses you might have the data aligned to the least significant bits instead. But is there a need to transport such data on parallel buses, at least so it would be expressed in mbus formats?

I have a parallel sensor that sends embedded data. I'll have a look how it organizes the data.

author maintainer
#24 lore raw

Re: [PATCH v3 04/10] media: uapi: Add generic serial metadata mbus formats

In reply to Laurent Pinchart

Hi Laurent, Tomi,

Quoted 54 lines · Laurent Pinchart

Hi Tomi,

Quoted 51 lines · Tomi Valkeinen
Quoted 38 lines · Laurent Pinchart
Quoted 7 lines · Tomi Valkeinen
Quoted 6 lines · Sakari Ailus
Quoted 5 lines · Tomi Valkeinen
Quoted 4 lines · Sakari Ailus

Add generic serial metadata mbus formats. These formats describe data width and packing but not the content itself. The reason for specifying such formats is that the formats as such are fairly device specific but they are still handled by CSI-2 receiver drivers that should not be aware

Do we want these formats to be CSI-2-specific ?

Quoted 27 lines
Quoted 26 lines
Quoted 9 lines
Quoted 3 lines

of device specific formats. What makes generic metadata formats possible is that these formats are parsed by software only, after capturing the data to system memory.

If I'm not mistaken, the CSI-2 spec doesn't say much about embedded data, except that it may exist. Afaics, in CSI-2, the embedded data is split into "lines", although the amount of data can be different than in the video lines.

Quoted 5 lines

The CCS specs talks more about embedded data. Some of it is quite odd, like:
"The length of the embedded data line shall not exceed the length of the image data line. The embedded data line should have the same length as the image data line.". I think it means the embedded line can be shorter than image line, but not longer.

That's what it means, yes. The CCS also has means to obtain the actual length --- frame format descriptors.

Quoted 5 lines

CCS also says that an embedded line should use 0x07 as padding at the end of the line, if there's less data than the image line.

CCS also talks about how the embedded data would be packed, and in some cases that packing would be the same as for pixel data.

In fact the packing is the same as for pixel data: the CSI-2 does not really differentiate there.

Quoted 7 lines
Quoted 4 lines

If I understand the CCS spec right, the packing is not the same as for the pixel data. You can have RAW12 pixel data but 8-byte embedded data. But if you meant that the different packing style options are the same for pixel and embedded data, yes.

The idea is to allow CSI-2 receivers to unpack embedded data the same way as pixel data at the hardware level, and guarantee that no embedded

Yes, but my comment was to Sakari's "the packing is the same as for pixel data" comment. Afaics, there's nothing in CSI-2 or CCS that says the packing is the same for pixel data and embedded data.

Quoted 8 lines
Quoted 5 lines

In fact, the CSS says "By default, top-embedded data shall use the same data packing as the Visible pixel data with one embedded data byte per pixel", implying that it would not always be the case. It continues that for RAW16/RAW20/RAW24 pixels the embedded data could be sent with RAW8/RAW10/RAW12.

The spec if a bit ambiguous :-S "by default" implies that something else could be done, but it's not clear what the other options are.

The options for packing are detailed in the spec. For higher bit depths there are options (as detailed by Tomi above).

Quoted 5 lines
Quoted 3 lines

That was just for top-embedded data, and I couldn't right away see mention of other embedded data.

The specification also mentions PDAF data for instance.

The bottom PDAF data is vendor defined (as it's processed) so the vendor is free to do whatever there. Note that the better way to support PDAF is via embedded PDAF pixels, not via separate bottom embedded (PDAF) data. The bottom embedded data could also be different kinds of statistics.

Quoted 110 lines
Quoted 9 lines
Quoted 8 lines

data significant byte gets split across multiple samples. When unpacking RAW10 pixel data, the CSI-2 receiver will take 5 bytes and output 4 samples on a 10-bit wide bus. For embedded data, it can conceptually do that same, and output embedded data with one byte per sample, aligned on the MSB of the 10-bit bus. However, in practice, I think the CSI-2 receiver will simply receive the bytes and send them to the DMA engine, without unpacking and then repacking.

Quoted 3 lines

At least TI CAL CSI-2 RX can extract the data from the CSI-2 stream using different unpacking for pixel and embedded data, and then (optionally) pack it in different ways when writing to memory.

Does this mean that the CAL could take RAW10 embedded data and drop the padding bytes when writing to memory ?

Quoted 94 lines
Quoted 29 lines

This can also be viewed as away to allow CSI-2 transmitters to serialize pixel data and embedded data the same way. For a RAW10 sensor, for instance, the embedded data would be generated on an internal 10-bit wide bus, aligned to the MSB, and would go through CSI-2 packing the same way that pixels do. That's nice, but it's internal to the sensor, so we don't need to expose it to userspace on that side.

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

But I don't think these formats are generic. They're defined in CCS, so shouldn't the format be, e.g., MEDIA_BUS_FMT_META_CCS_RAW_12 or such?

The reason for having generic definitions is that we do not need receiver drivers to be aware of formats that are specific to another driver.

Quoted 8 lines

Yes, I agree with that, and that's not my point here. But "generic" doesn't mean the definitions are not for some particular bus or standard, "generic" just means that it doesn't specify the content, only the packing.

My point is that these packings seem to be specific to CCS. While non-CCS compliant sensors may use the same packing, the packing itself is still a "CCS packing".

Agreed.

Quoted 4 lines

So why not call them that? The 8-bit format is
fully generic, whereas the rest are CCS packings, as defined in the CCS spec (the CCS spec also specifies the content, but here were only using the CCS packing).

Quoted 9 lines

I think an additional question is whether we actually need multiple media bus codes here. Are there CSI-2 receivers, or other downstream blocks, that will need to be configured differently to receive embedded data from a CCS (in the sense of using CCS packing, not being fully CCS-compliant) RAW10 sensor compared to a RAW12 sensor ? Or will the CSI-2 receiver just send bytes to memory without caring ? In the latter case, it will be the application that will need to know how the data is packed to ignore one byte out of three for RAW12 and out of five for RAW10.

I would presume all receivers can operate in "just write what you receive" mode, in which case the packing doesn't matter as you say, but then again, I think it's safer to have mbus codes for the different packings.

Quoted 34 lines
Quoted 10 lines

Maybe they shouldn't be called "generic", but... umm... Content unaware metadata formats... doesn't sound very good =).

Also, I'm a bit confused about CSI-2 pixel and embedded data formats and how we handle them.

For MEDIA_BUS_FMT_SBGGR10_1X10, we define that the data contains 10 bits per pixel, from bit 0 to 9. But CSI-2, for RAW10, actually sends it differently, with the higher bits first, and the lowest bits in the fourth byte. So that CSI-2 packing is implicit, kind of hidden here.

Usage of MEDIA_BUS_FMT_SBGGR10_1X10 for CSI-2 is a historical decision (or mistake, depending on how you look at it). It doesn't match what actually happens on the bus.

Quoted 9 lines

Which is probably fine, as we're really only interested in the unpacked data, not the CSI-2 packed. And when writing this data to memory, the DMA engine can write it either as is, or unpack each pixel to a 16-bit container.

For MEDIA_BUS_FMT_META_10, we define it similarly to MEDIA_BUS_FMT_SBGGR10_1X10, except the lowest 2 bits are marked as padding bits. And, I think, MEDIA_BUS_FMT_META_10 implies RAW10 CSI-2 format.

That's how I understand it too

Quoted 6 lines

However, when writing the data to memory, we don't want either of the modes used in the above pixel data case, but rather we want to write the data as it is in the CSI-2 bus. So, the DMA engine can either reverse the RAW10 unpacking to get the wanted output format, or alternatively the CSI-2 receiver could instead use RAW8 mode to avoid any unpacking.

Quoted 5 lines

I don't think the DMA engine would "reverse the RAW10 packing" here. It will write bytes as they arrive (or more likely, a set of bytes at a time to operate on bursts). The other option would be for the DMA engine to drop the padding bytes, but I don't know of any CSI-2 receiver that can do that.

Well, for pixel data, TI CAL has the following options when writing to memory for, e.g. RAW12:
- RAW12 MIPI (a separate byte for lsbs)
- RAW12 Linear (12 consecutive bits per pixel)
- RAW16 (12 consecutive bits per pixel in a 16bit container)

Of these, only the RAW12 MIPI makes sense for embedded data (actually the whole processing block can be and should be skipped for embedded data, so it's more like CAL is receiving RAW8 and writing out RAW8).

Quoted 8 lines
Quoted 4 lines

Or, if it would be possible, skipping the padding bytes. Which looks like it is almost possible with TI CAL, but not quite: it can write with RAW8, but afaics that always takes the lowest bits of the 12-bit pixel, but here we'd like it to take the highest bits.

That's a weird one, even for image data, dropping the MSBs doesn't make much sense, while writing the RAW8 MSB when receiving RAW10 or RAW12 would be useful.

I'd suppose it would drop the four most significant bits if it was a parallel receiver. Does CAL have 8-bit input support? That should just work in this case.

Signature · 3 lines
Regards,

Sakari Ailus
maintainer
#25 lore raw

Re: [PATCH v3 04/10] media: uapi: Add generic serial metadata mbus formats

Hi Sakari,

Quoted 59 lines · Sakari Ailus
Quoted 58 lines · Laurent Pinchart
Quoted 56 lines · Tomi Valkeinen
Quoted 38 lines · Laurent Pinchart
Quoted 7 lines · Tomi Valkeinen
Quoted 6 lines · Sakari Ailus
Quoted 5 lines · Tomi Valkeinen
Quoted 4 lines · Sakari Ailus

Add generic serial metadata mbus formats. These formats describe data width and packing but not the content itself. The reason for specifying such formats is that the formats as such are fairly device specific but they are still handled by CSI-2 receiver drivers that should not be aware

Do we want these formats to be CSI-2-specific ?

Quoted 27 lines
Quoted 26 lines
Quoted 9 lines
Quoted 3 lines

of device specific formats. What makes generic metadata formats possible is that these formats are parsed by software only, after capturing the data to system memory.

If I'm not mistaken, the CSI-2 spec doesn't say much about embedded data, except that it may exist. Afaics, in CSI-2, the embedded data is split into "lines", although the amount of data can be different than in the video lines.

Quoted 5 lines

The CCS specs talks more about embedded data. Some of it is quite odd, like:
"The length of the embedded data line shall not exceed the length of the image data line. The embedded data line should have the same length as the image data line.". I think it means the embedded line can be shorter than image line, but not longer.

That's what it means, yes. The CCS also has means to obtain the actual length --- frame format descriptors.

Quoted 5 lines

CCS also says that an embedded line should use 0x07 as padding at the end of the line, if there's less data than the image line.

CCS also talks about how the embedded data would be packed, and in some cases that packing would be the same as for pixel data.

In fact the packing is the same as for pixel data: the CSI-2 does not really differentiate there.

Quoted 7 lines
Quoted 4 lines

If I understand the CCS spec right, the packing is not the same as for the pixel data. You can have RAW12 pixel data but 8-byte embedded data. But if you meant that the different packing style options are the same for pixel and embedded data, yes.

The idea is to allow CSI-2 receivers to unpack embedded data the same way as pixel data at the hardware level, and guarantee that no embedded

Yes, but my comment was to Sakari's "the packing is the same as for pixel data" comment. Afaics, there's nothing in CSI-2 or CCS that says the packing is the same for pixel data and embedded data.

In fact, the CSS says "By default, top-embedded data shall use the same data packing as the Visible pixel data with one embedded data byte per pixel", implying that it would not always be the case. It continues that for RAW16/RAW20/RAW24 pixels the embedded data could be sent with RAW8/RAW10/RAW12.

Quoted 5 lines
Quoted 2 lines

The spec if a bit ambiguous :-S "by default" implies that something else could be done, but it's not clear what the other options are.

The options for packing are detailed in the spec. For higher bit depths there are options (as detailed by Tomi above).

The "by default" wording could be construed as allowing any packing. I think the intent was to only allow an alternative packing for RAW16 and higher, but it's not very clear.

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

That was just for top-embedded data, and I couldn't right away see mention of other embedded data.

The specification also mentions PDAF data for instance.

The bottom PDAF data is vendor defined (as it's processed) so the vendor is free to do whatever there. Note that the better way to support PDAF is via embedded PDAF pixels, not via separate bottom embedded (PDAF) data. The bottom embedded data could also be different kinds of statistics.

Lots of ISPs don't support hardware processing of embedded PDAF pixels, so sensor-side processing is often nice to have. I wouldn't consider the embedded PDAF lines to be always worse than embedded PDAF pixels.

Quoted 121 lines
Quoted 109 lines
Quoted 11 lines
Quoted 7 lines

data significant byte gets split across multiple samples. When unpacking RAW10 pixel data, the CSI-2 receiver will take 5 bytes and output 4 samples on a 10-bit wide bus. For embedded data, it can conceptually do that same, and output embedded data with one byte per sample, aligned on the MSB of the 10-bit bus. However, in practice, I think the CSI-2 receiver will simply receive the bytes and send them to the DMA engine, without unpacking and then repacking.

At least TI CAL CSI-2 RX can extract the data from the CSI-2 stream using different unpacking for pixel and embedded data, and then (optionally) pack it in different ways when writing to memory.

Does this mean that the CAL could take RAW10 embedded data and drop the padding bytes when writing to memory ?

Quoted 94 lines
Quoted 29 lines

This can also be viewed as away to allow CSI-2 transmitters to serialize pixel data and embedded data the same way. For a RAW10 sensor, for instance, the embedded data would be generated on an internal 10-bit wide bus, aligned to the MSB, and would go through CSI-2 packing the same way that pixels do. That's nice, but it's internal to the sensor, so we don't need to expose it to userspace on that side.

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

But I don't think these formats are generic. They're defined in CCS, so shouldn't the format be, e.g., MEDIA_BUS_FMT_META_CCS_RAW_12 or such?

The reason for having generic definitions is that we do not need receiver drivers to be aware of formats that are specific to another driver.

Quoted 8 lines

Yes, I agree with that, and that's not my point here. But "generic" doesn't mean the definitions are not for some particular bus or standard, "generic" just means that it doesn't specify the content, only the packing.

My point is that these packings seem to be specific to CCS. While non-CCS compliant sensors may use the same packing, the packing itself is still a "CCS packing".

Agreed.

Quoted 4 lines

So why not call them that? The 8-bit format is
fully generic, whereas the rest are CCS packings, as defined in the CCS spec (the CCS spec also specifies the content, but here were only using the CCS packing).

Quoted 9 lines

I think an additional question is whether we actually need multiple media bus codes here. Are there CSI-2 receivers, or other downstream blocks, that will need to be configured differently to receive embedded data from a CCS (in the sense of using CCS packing, not being fully CCS-compliant) RAW10 sensor compared to a RAW12 sensor ? Or will the CSI-2 receiver just send bytes to memory without caring ? In the latter case, it will be the application that will need to know how the data is packed to ignore one byte out of three for RAW12 and out of five for RAW10.

I would presume all receivers can operate in "just write what you receive" mode, in which case the packing doesn't matter as you say, but then again, I think it's safer to have mbus codes for the different packings.

Quoted 34 lines
Quoted 10 lines

Maybe they shouldn't be called "generic", but... umm... Content unaware metadata formats... doesn't sound very good =).

Also, I'm a bit confused about CSI-2 pixel and embedded data formats and how we handle them.

For MEDIA_BUS_FMT_SBGGR10_1X10, we define that the data contains 10 bits per pixel, from bit 0 to 9. But CSI-2, for RAW10, actually sends it differently, with the higher bits first, and the lowest bits in the fourth byte. So that CSI-2 packing is implicit, kind of hidden here.

Usage of MEDIA_BUS_FMT_SBGGR10_1X10 for CSI-2 is a historical decision (or mistake, depending on how you look at it). It doesn't match what actually happens on the bus.

Quoted 9 lines

Which is probably fine, as we're really only interested in the unpacked data, not the CSI-2 packed. And when writing this data to memory, the DMA engine can write it either as is, or unpack each pixel to a 16-bit container.

For MEDIA_BUS_FMT_META_10, we define it similarly to MEDIA_BUS_FMT_SBGGR10_1X10, except the lowest 2 bits are marked as padding bits. And, I think, MEDIA_BUS_FMT_META_10 implies RAW10 CSI-2 format.

That's how I understand it too

Quoted 6 lines

However, when writing the data to memory, we don't want either of the modes used in the above pixel data case, but rather we want to write the data as it is in the CSI-2 bus. So, the DMA engine can either reverse the RAW10 unpacking to get the wanted output format, or alternatively the CSI-2 receiver could instead use RAW8 mode to avoid any unpacking.

Quoted 5 lines

I don't think the DMA engine would "reverse the RAW10 packing" here. It will write bytes as they arrive (or more likely, a set of bytes at a time to operate on bursts). The other option would be for the DMA engine to drop the padding bytes, but I don't know of any CSI-2 receiver that can do that.

Well, for pixel data, TI CAL has the following options when writing to memory for, e.g. RAW12:
- RAW12 MIPI (a separate byte for lsbs)
- RAW12 Linear (12 consecutive bits per pixel)
- RAW16 (12 consecutive bits per pixel in a 16bit container)

Of these, only the RAW12 MIPI makes sense for embedded data (actually the whole processing block can be and should be skipped for embedded data, so it's more like CAL is receiving RAW8 and writing out RAW8).

Quoted 8 lines
Quoted 4 lines

Or, if it would be possible, skipping the padding bytes. Which looks like it is almost possible with TI CAL, but not quite: it can write with RAW8, but afaics that always takes the lowest bits of the 12-bit pixel, but here we'd like it to take the highest bits.

That's a weird one, even for image data, dropping the MSBs doesn't make much sense, while writing the RAW8 MSB when receiving RAW10 or RAW12 would be useful.

I'd suppose it would drop the four most significant bits if it was a parallel receiver. Does CAL have 8-bit input support? That should just work in this case.

Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#26 lore raw

Re: [PATCH v3 04/10] media: uapi: Add generic serial metadata mbus formats

Hi Laurent,

Quoted 65 lines · Laurent Pinchart

Hi Sakari,

Quoted 62 lines · Sakari Ailus
Quoted 60 lines · Laurent Pinchart
Quoted 51 lines · Tomi Valkeinen
Quoted 38 lines · Laurent Pinchart
Quoted 7 lines · Tomi Valkeinen
Quoted 6 lines · Sakari Ailus
Quoted 5 lines · Tomi Valkeinen
Quoted 4 lines · Sakari Ailus

Add generic serial metadata mbus formats. These formats describe data width and packing but not the content itself. The reason for specifying such formats is that the formats as such are fairly device specific but they are still handled by CSI-2 receiver drivers that should not be aware

Do we want these formats to be CSI-2-specific ?

Quoted 27 lines
Quoted 26 lines
Quoted 9 lines
Quoted 3 lines

of device specific formats. What makes generic metadata formats possible is that these formats are parsed by software only, after capturing the data to system memory.

If I'm not mistaken, the CSI-2 spec doesn't say much about embedded data, except that it may exist. Afaics, in CSI-2, the embedded data is split into "lines", although the amount of data can be different than in the video lines.

Quoted 5 lines

The CCS specs talks more about embedded data. Some of it is quite odd, like:
"The length of the embedded data line shall not exceed the length of the image data line. The embedded data line should have the same length as the image data line.". I think it means the embedded line can be shorter than image line, but not longer.

That's what it means, yes. The CCS also has means to obtain the actual length --- frame format descriptors.

Quoted 5 lines

CCS also says that an embedded line should use 0x07 as padding at the end of the line, if there's less data than the image line.

CCS also talks about how the embedded data would be packed, and in some cases that packing would be the same as for pixel data.

In fact the packing is the same as for pixel data: the CSI-2 does not really differentiate there.

Quoted 7 lines
Quoted 4 lines

If I understand the CCS spec right, the packing is not the same as for the pixel data. You can have RAW12 pixel data but 8-byte embedded data. But if you meant that the different packing style options are the same for pixel and embedded data, yes.

The idea is to allow CSI-2 receivers to unpack embedded data the same way as pixel data at the hardware level, and guarantee that no embedded

Yes, but my comment was to Sakari's "the packing is the same as for pixel data" comment. Afaics, there's nothing in CSI-2 or CCS that says the packing is the same for pixel data and embedded data.

Quoted 5 lines

In fact, the CSS says "By default, top-embedded data shall use the same data packing as the Visible pixel data with one embedded data byte per pixel", implying that it would not always be the case. It continues that for RAW16/RAW20/RAW24 pixels the embedded data could be sent with RAW8/RAW10/RAW12.

The spec if a bit ambiguous :-S "by default" implies that something else could be done, but it's not clear what the other options are.

Quoted 6 lines
Quoted 2 lines

The options for packing are detailed in the spec. For higher bit depths there are options (as detailed by Tomi above).

The "by default" wording could be construed as allowing any packing. I think the intent was to only allow an alternative packing for RAW16 and higher, but it's not very clear.

The optimised packing convention is only available for RAW16 and higher --- otherwise you won't have enough bits needed to carry two bytes of data. The packing used is configured using emb_data_ctrl register.

Maybe annex A could be merged to section 7.15 in order to make things clearer.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

That was just for top-embedded data, and I couldn't right away see mention of other embedded data.

The specification also mentions PDAF data for instance.

Quoted 8 lines
Quoted 4 lines

The bottom PDAF data is vendor defined (as it's processed) so the vendor is free to do whatever there. Note that the better way to support PDAF is via embedded PDAF pixels, not via separate bottom embedded (PDAF) data. The bottom embedded data could also be different kinds of statistics.

Lots of ISPs don't support hardware processing of embedded PDAF pixels, so sensor-side processing is often nice to have. I wouldn't consider the embedded PDAF lines to be always worse than embedded PDAF pixels.

Not necessarily, indeed, but these formats tend not to be documented. :-(

Signature · 3 lines
Regards,

Sakari Ailus
author maintainer
#27 lore raw

PATCH v3 5/10 media: uapi: Document which mbus format fields are valid for metadata

In reply to Sakari Ailus

Now that metadata mbus formats have been added, it is necessary to define which fields in struct v4l2_mbus_format are applicable to them (not many).

---
 include/uapi/linux/v4l2-mediabus.h | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)
include/uapi/linux/v4l2-mediabus.h +12 -6
index 6b07b73473b5..3cadb3b58b85 100644
@@ -19,12 +19,18 @@
  * @width:	image width
  * @height:	image height
  * @code:	data format code (from enum v4l2_mbus_pixelcode)
- * @field:	used interlacing type (from enum v4l2_field)
- * @colorspace:	colorspace of the data (from enum v4l2_colorspace)
- * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding)
- * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding)
- * @quantization: quantization of the data (from enum v4l2_quantization)
- * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func)
+ * @field:	used interlacing type (from enum v4l2_field), not applicable
+ *		to metadata mbus codes
+ * @colorspace:	colorspace of the data (from enum v4l2_colorspace), zero on
+ *		metadata mbus codes
+ * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding), zero
+ *		on metadata mbus codes
+ * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding), zero on
+ *		metadata mbus codes
+ * @quantization: quantization of the data (from enum v4l2_quantization), zero
+ *		on metadata mbus codes
+ * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func), zero
+ *		on metadata mbus codes
  * @flags:	flags (V4L2_MBUS_FRAMEFMT_*)
  * @reserved:  reserved bytes that can be later used
  */
Signature · 1 line
2.39.2
#28 lore raw

Re: [PATCH v3 05/10] media: uapi: Document which mbus format fields are valid for metadata

Hi Sakari

Quoted 14 lines · Sakari Ailus

Now that metadata mbus formats have been added, it is necessary to define which fields in struct v4l2_mbus_format are applicable to them (not many).

---
 include/uapi/linux/v4l2-mediabus.h | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)
include/uapi/linux/v4l2-mediabus.h +0 -0
index 6b07b73473b5..3cadb3b58b85 100644
@@ -19,12 +19,18 @@
  * @width:	image width

include/uapi/linux/v4l2-mediabus.h line 23

Quoted 10 lines
include/uapi/linux/v4l2-mediabus.h +2 -6
  * @height:	image height
  * @code:	data format code (from enum v4l2_mbus_pixelcode)
- * @field:	used interlacing type (from enum v4l2_field)
- * @colorspace:	colorspace of the data (from enum v4l2_colorspace)
- * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding)
- * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding)
- * @quantization: quantization of the data (from enum v4l2_quantization)
- * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func)
+ * @field:	used interlacing type (from enum v4l2_field), not applicable
+ *		to metadata mbus codes

"not applicable" is a bit geeric. Should this be set to V4L2_FIELD_NONE (for metadata, and progressive image formats maybe ?)

include/uapi/linux/v4l2-mediabus.h line 29

Quoted 6 lines
include/uapi/linux/v4l2-mediabus.h +6 -0
+ * @colorspace:	colorspace of the data (from enum v4l2_colorspace), zero on
+ *		metadata mbus codes
+ * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding), zero
+ *		on metadata mbus codes
+ * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding), zero on
+ *		metadata mbus codes

Can this be zero ?

enum v4l2_hsv_encoding {

	/* Hue mapped to 0 - 179 */
	V4L2_HSV_ENC_180		= 128,

	/* Hue mapped to 0-255 */
	V4L2_HSV_ENC_256		= 129,
};
Quoted 10 lines
include/uapi/linux/v4l2-mediabus.h +4 -1
+ * @quantization: quantization of the data (from enum v4l2_quantization), zero
+ *		on metadata mbus codes
+ * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func), zero
+ *		on metadata mbus codes
  * @flags:	flags (V4L2_MBUS_FRAMEFMT_*)
  * @reserved:  reserved bytes that can be later used
  */
--

2.39.2

author maintainer
#29 lore raw

Re: [PATCH v3 05/10] media: uapi: Document which mbus format fields are valid for metadata

Hi Jacopo,

Thanks for the review.

Quoted 22 lines · Jacopo Mondi

Hi Sakari

Quoted 19 lines · Sakari Ailus

Now that metadata mbus formats have been added, it is necessary to define which fields in struct v4l2_mbus_format are applicable to them (not many).

---
 include/uapi/linux/v4l2-mediabus.h | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)
include/uapi/linux/v4l2-mediabus.h +0 -3
index 6b07b73473b5..3cadb3b58b85 100644
@@ -19,12 +19,18 @@
  * @width:	image width
  * @height:	image height
  * @code:	data format code (from enum v4l2_mbus_pixelcode)
- * @field:	used interlacing type (from enum v4l2_field)
- * @colorspace:	colorspace of the data (from enum v4l2_colorspace)
- * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding)
Quoted 8 lines

include/uapi/linux/v4l2-mediabus.h line 23

Quoted 5 lines
include/uapi/linux/v4l2-mediabus.h +2 -3
- * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding)
- * @quantization: quantization of the data (from enum v4l2_quantization)
- * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func)
+ * @field:	used interlacing type (from enum v4l2_field), not applicable
+ *		to metadata mbus codes

"not applicable" is a bit geeric. Should this be set to V4L2_FIELD_NONE (for metadata, and progressive image formats maybe ?)

I actually intended to have the same wording here than for the other fields but missed changing this.

0 corresponds to V4L2_FIELD_ANY.

Quoted 10 lines

include/uapi/linux/v4l2-mediabus.h line 29

Quoted 7 lines
include/uapi/linux/v4l2-mediabus.h +6 -0
+ * @colorspace:	colorspace of the data (from enum v4l2_colorspace), zero on
+ *		metadata mbus codes
+ * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding), zero
+ *		on metadata mbus codes
+ * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding), zero on
+ *		metadata mbus codes

Can this be zero ?

Quoted 8 lines

enum v4l2_hsv_encoding {

	/* Hue mapped to 0 - 179 */
	V4L2_HSV_ENC_180		= 128,

	/* Hue mapped to 0-255 */
	V4L2_HSV_ENC_256		= 129,
};

Good question. Neither value is meaningful for metadata.

The values appear to be such in the enum to avoid colliding with ycbcr encoding values (another field above).

Generally, what doesn't matter will be zero. These fields have been added at some point and a lot of drivers do not set them, even for pixel data.

I wonder what Hans and Laurent think.

Quoted 8 lines
Quoted 8 lines
include/uapi/linux/v4l2-mediabus.h +4 -0
+ * @quantization: quantization of the data (from enum v4l2_quantization), zero
+ *		on metadata mbus codes
+ * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func), zero
+ *		on metadata mbus codes
  * @flags:	flags (V4L2_MBUS_FRAMEFMT_*)
  * @reserved:  reserved bytes that can be later used
  */
Signature · 3 lines
Kind regards,

Sakari Ailus
maintainer
#30 lore raw

Re: [PATCH v3 05/10] media: uapi: Document which mbus format fields are valid for metadata

Quoted 55 lines · Sakari Ailus
Quoted 20 lines · Jacopo Mondi
Quoted 19 lines · Sakari Ailus

Now that metadata mbus formats have been added, it is necessary to define which fields in struct v4l2_mbus_format are applicable to them (not many).

---
 include/uapi/linux/v4l2-mediabus.h | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)
include/uapi/linux/v4l2-mediabus.h +0 -3
index 6b07b73473b5..3cadb3b58b85 100644
@@ -19,12 +19,18 @@
  * @width:	image width
  * @height:	image height
  * @code:	data format code (from enum v4l2_mbus_pixelcode)
- * @field:	used interlacing type (from enum v4l2_field)
- * @colorspace:	colorspace of the data (from enum v4l2_colorspace)
- * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding)
Quoted 8 lines

include/uapi/linux/v4l2-mediabus.h line 23

Quoted 5 lines
include/uapi/linux/v4l2-mediabus.h +2 -3
- * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding)
- * @quantization: quantization of the data (from enum v4l2_quantization)
- * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func)
+ * @field:	used interlacing type (from enum v4l2_field), not applicable
+ *		to metadata mbus codes

"not applicable" is a bit geeric. Should this be set to V4L2_FIELD_NONE (for metadata, and progressive image formats maybe ?)

I actually intended to have the same wording here than for the other fields but missed changing this.

0 corresponds to V4L2_FIELD_ANY.

Quoted 9 lines

include/uapi/linux/v4l2-mediabus.h line 29

Quoted 6 lines
include/uapi/linux/v4l2-mediabus.h +6 -0
+ * @colorspace:	colorspace of the data (from enum v4l2_colorspace), zero on
+ *		metadata mbus codes
+ * @ycbcr_enc:	YCbCr encoding of the data (from enum v4l2_ycbcr_encoding), zero
+ *		on metadata mbus codes
+ * @hsv_enc:	HSV encoding of the data (from enum v4l2_hsv_encoding), zero on
+ *		metadata mbus codes

Can this be zero ?

Quoted 8 lines

enum v4l2_hsv_encoding {

	/* Hue mapped to 0 - 179 */
	V4L2_HSV_ENC_180		= 128,

	/* Hue mapped to 0-255 */
	V4L2_HSV_ENC_256		= 129,
};

Good question. Neither value is meaningful for metadata.

Quoted 7 lines

The values appear to be such in the enum to avoid colliding with ycbcr encoding values (another field above).

Generally, what doesn't matter will be zero. These fields have been added at some point and a lot of drivers do not set them, even for pixel data.

I wonder what Hans and Laurent think.

ycbcr_enc and hsv_enc are stored in an unnamed union.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
include/uapi/linux/v4l2-mediabus.h +4 -0
+ * @quantization: quantization of the data (from enum v4l2_quantization), zero
+ *		on metadata mbus codes
+ * @xfer_func:  transfer function of the data (from enum v4l2_xfer_func), zero
+ *		on metadata mbus codes
  * @flags:	flags (V4L2_MBUS_FRAMEFMT_*)
  * @reserved:  reserved bytes that can be later used
  */
Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#31 lore raw

PATCH v3 6/10 media: uapi: Add a macro to tell whether an mbus code is metadata

In reply to Sakari Ailus

Add a macro to tell whether a given mbus code is metadata.

---
 include/uapi/linux/media-bus-format.h | 3 +++
 1 file changed, 3 insertions(+)
include/uapi/linux/media-bus-format.h +3 -0
index 9ee031397372..2486b4178c5f 100644
@@ -182,4 +182,7 @@
 #define MEDIA_BUS_FMT_META_20			0x8006
 #define MEDIA_BUS_FMT_META_24			0x8007
 
+#define MEDIA_BUS_FMT_IS_META(code)		\
+	((code) & 0xf000 == 0x7000 || (code) & 0xf000 == 0x8000)
+
 #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
Signature · 1 line
2.39.2
#32 lore raw

Re: [PATCH v3 06/10] media: uapi: Add a macro to tell whether an mbus code is metadata

Quoted 12 lines · Sakari Ailus

Add a macro to tell whether a given mbus code is metadata.

---
 include/uapi/linux/media-bus-format.h | 3 +++
 1 file changed, 3 insertions(+)
include/uapi/linux/media-bus-format.h +0 -0
index 9ee031397372..2486b4178c5f 100644

include/uapi/linux/media-bus-format.h line 186

Quoted 5 lines
include/uapi/linux/media-bus-format.h +2 -0
@@ -182,4 +182,7 @@
 #define MEDIA_BUS_FMT_META_20			0x8006
 #define MEDIA_BUS_FMT_META_24			0x8007
 
+#define MEDIA_BUS_FMT_IS_META(code)		\
+	((code) & 0xf000 == 0x7000 || (code) & 0xf000 == 0x8000)

I don't think this is right: == has a higher precedence than &, so this actually reads:

	code & (0xf000 == 0x7000)

Regards,

Hans

Quoted 2 lines
include/uapi/linux/media-bus-format.h +1 -0
+
 #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
author maintainer
#33 lore raw

Re: [PATCH v3 06/10] media: uapi: Add a macro to tell whether an mbus code is metadata

Quoted 16 lines · Hans Verkuil
Quoted 15 lines · Sakari Ailus

Add a macro to tell whether a given mbus code is metadata.

---
 include/uapi/linux/media-bus-format.h | 3 +++
 1 file changed, 3 insertions(+)
include/uapi/linux/media-bus-format.h +0 -0
index 9ee031397372..2486b4178c5f 100644
@@ -182,4 +182,7 @@
 #define MEDIA_BUS_FMT_META_20			0x8006
 #define MEDIA_BUS_FMT_META_24			0x8007
 
Quoted 7 lines

include/uapi/linux/media-bus-format.h line 186

Quoted 2 lines
include/uapi/linux/media-bus-format.h +2 -0
+#define MEDIA_BUS_FMT_IS_META(code)		\
+	((code) & 0xf000 == 0x7000 || (code) & 0xf000 == 0x8000)

I don't think this is right: == has a higher precedence than &, so this actually reads:

	code & (0xf000 == 0x7000)

You're right, I'll fix this for v4.

Signature · 1 line
Sakari Ailus
#34 lore raw

Re: [PATCH v3 06/10] media: uapi: Add a macro to tell whether an mbus code is metadata

In reply to Sakari Ailus

Hi,

Quoted 12 lines · Sakari Ailus

Add a macro to tell whether a given mbus code is metadata.

---
  include/uapi/linux/media-bus-format.h | 3 +++
  1 file changed, 3 insertions(+)
include/uapi/linux/media-bus-format.h +0 -0
index 9ee031397372..2486b4178c5f 100644

include/uapi/linux/media-bus-format.h line 188

Quoted 7 lines
include/uapi/linux/media-bus-format.h +3 -0
@@ -182,4 +182,7 @@
  #define MEDIA_BUS_FMT_META_20			0x8006
  #define MEDIA_BUS_FMT_META_24			0x8007
 
+#define MEDIA_BUS_FMT_IS_META(code)		\
+	((code) & 0xf000 == 0x7000 || (code) & 0xf000 == 0x8000)
+
  #endif /* __LINUX_MEDIA_BUS_FORMAT_H */

mbus code seems to be u32, so the above won't work. Maybe:

(((code) & ~0xfffULL) == 0x7000 || ((code) & ~0xfffULL) == 0x8000)

Also, embedded formats with 0x9nnn codes are added later in the series.

Tomi

author maintainer
#35 lore raw

Re: [PATCH v3 06/10] media: uapi: Add a macro to tell whether an mbus code is metadata

Moi,

Quoted 24 lines · Tomi Valkeinen

Hi,

Quoted 12 lines · Sakari Ailus

Add a macro to tell whether a given mbus code is metadata.

---
  include/uapi/linux/media-bus-format.h | 3 +++
  1 file changed, 3 insertions(+)
include/uapi/linux/media-bus-format.h +0 -0
index 9ee031397372..2486b4178c5f 100644

include/uapi/linux/media-bus-format.h line 187

Quoted 6 lines
include/uapi/linux/media-bus-format.h +3 -0
@@ -182,4 +182,7 @@
  #define MEDIA_BUS_FMT_META_20			0x8006
  #define MEDIA_BUS_FMT_META_24			0x8007
+#define MEDIA_BUS_FMT_IS_META(code)		\
+	((code) & 0xf000 == 0x7000 || (code) & 0xf000 == 0x8000)
+
  #endif /* __LINUX_MEDIA_BUS_FORMAT_H */

mbus code seems to be u32, so the above won't work. Maybe:

Quoted 3 lines

(((code) & ~0xfffULL) == 0x7000 || ((code) & ~0xfffULL) == 0x8000)

Also, embedded formats with 0x9nnn codes are added later in the series.

Thanks, I'll address these for v4.

Signature · 1 line
Sakari Ailus
maintainer
#36 lore raw

Re: [PATCH v3 06/10] media: uapi: Add a macro to tell whether an mbus code is metadata

Quoted 23 lines · Sakari Ailus
Quoted 22 lines · Tomi Valkeinen
Quoted 12 lines · Sakari Ailus

Add a macro to tell whether a given mbus code is metadata.

---
  include/uapi/linux/media-bus-format.h | 3 +++
  1 file changed, 3 insertions(+)
include/uapi/linux/media-bus-format.h +0 -0
index 9ee031397372..2486b4178c5f 100644

include/uapi/linux/media-bus-format.h line 187

Quoted 6 lines
include/uapi/linux/media-bus-format.h +3 -0
@@ -182,4 +182,7 @@
  #define MEDIA_BUS_FMT_META_20			0x8006
  #define MEDIA_BUS_FMT_META_24			0x8007
+#define MEDIA_BUS_FMT_IS_META(code)		\
+	((code) & 0xf000 == 0x7000 || (code) & 0xf000 == 0x8000)
+
  #endif /* __LINUX_MEDIA_BUS_FORMAT_H */

mbus code seems to be u32, so the above won't work. Maybe:

Quoted 5 lines
Quoted 3 lines

(((code) & ~0xfffULL) == 0x7000 || ((code) & ~0xfffULL) == 0x8000)

Also, embedded formats with 0x9nnn codes are added later in the series.

Thanks, I'll address these for v4.

It would be nice to make this an inline function, to avoid evaluating the code twice. I think you can move it to an internal kernel header, it doesn't need to be exposed to userspace.

Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#37 lore raw

Re: [PATCH v3 06/10] media: uapi: Add a macro to tell whether an mbus code is metadata

Hi Laurent,

Quoted 26 lines · Laurent Pinchart
Quoted 25 lines · Sakari Ailus
Quoted 24 lines · Tomi Valkeinen
Quoted 12 lines · Sakari Ailus

Add a macro to tell whether a given mbus code is metadata.

---
  include/uapi/linux/media-bus-format.h | 3 +++
  1 file changed, 3 insertions(+)
include/uapi/linux/media-bus-format.h +0 -0
index 9ee031397372..2486b4178c5f 100644

include/uapi/linux/media-bus-format.h line 187

Quoted 6 lines
include/uapi/linux/media-bus-format.h +3 -0
@@ -182,4 +182,7 @@
  #define MEDIA_BUS_FMT_META_20			0x8006
  #define MEDIA_BUS_FMT_META_24			0x8007
+#define MEDIA_BUS_FMT_IS_META(code)		\
+	((code) & 0xf000 == 0x7000 || (code) & 0xf000 == 0x8000)
+
  #endif /* __LINUX_MEDIA_BUS_FORMAT_H */

mbus code seems to be u32, so the above won't work. Maybe:

(((code) & ~0xfffULL) == 0x7000 || ((code) & ~0xfffULL) == 0x8000)

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Also, embedded formats with 0x9nnn codes are added later in the series.

Thanks, I'll address these for v4.

It would be nice to make this an inline function, to avoid evaluating the code twice. I think you can move it to an internal kernel header, it doesn't need to be exposed to userspace.

This was meant for user space consumption for setting the buffer type to the video node accordingly. It's certainly possible to do that without this macro, too.

Signature · 3 lines
Regards,

Sakari Ailus
maintainer
#38 lore raw

Re: [PATCH v3 06/10] media: uapi: Add a macro to tell whether an mbus code is metadata

Quoted 31 lines · Sakari Ailus
Quoted 30 lines · Laurent Pinchart
Quoted 28 lines · Sakari Ailus
Quoted 22 lines · Tomi Valkeinen
Quoted 12 lines · Sakari Ailus

Add a macro to tell whether a given mbus code is metadata.

---
  include/uapi/linux/media-bus-format.h | 3 +++
  1 file changed, 3 insertions(+)
include/uapi/linux/media-bus-format.h +0 -0
index 9ee031397372..2486b4178c5f 100644

include/uapi/linux/media-bus-format.h line 187

Quoted 6 lines
include/uapi/linux/media-bus-format.h +3 -0
@@ -182,4 +182,7 @@
  #define MEDIA_BUS_FMT_META_20			0x8006
  #define MEDIA_BUS_FMT_META_24			0x8007
+#define MEDIA_BUS_FMT_IS_META(code)		\
+	((code) & 0xf000 == 0x7000 || (code) & 0xf000 == 0x8000)
+
  #endif /* __LINUX_MEDIA_BUS_FORMAT_H */

mbus code seems to be u32, so the above won't work. Maybe:

Quoted 3 lines

(((code) & ~0xfffULL) == 0x7000 || ((code) & ~0xfffULL) == 0x8000)

Also, embedded formats with 0x9nnn codes are added later in the series.

Thanks, I'll address these for v4.

Quoted 7 lines
Quoted 3 lines

It would be nice to make this an inline function, to avoid evaluating the code twice. I think you can move it to an internal kernel header, it doesn't need to be exposed to userspace.

This was meant for user space consumption for setting the buffer type to the video node accordingly. It's certainly possible to do that without this macro, too.

I think userspace would be fine without it :-)

Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#39 lore raw

Re: [PATCH v3 06/10] media: uapi: Add a macro to tell whether an mbus code is metadata

Quoted 36 lines · Laurent Pinchart
Quoted 35 lines · Sakari Ailus
Quoted 33 lines · Laurent Pinchart
Quoted 23 lines · Sakari Ailus
Quoted 22 lines · Tomi Valkeinen
Quoted 12 lines · Sakari Ailus

Add a macro to tell whether a given mbus code is metadata.

---
  include/uapi/linux/media-bus-format.h | 3 +++
  1 file changed, 3 insertions(+)
include/uapi/linux/media-bus-format.h +0 -0
index 9ee031397372..2486b4178c5f 100644

include/uapi/linux/media-bus-format.h line 187

Quoted 6 lines
include/uapi/linux/media-bus-format.h +3 -0
@@ -182,4 +182,7 @@
  #define MEDIA_BUS_FMT_META_20			0x8006
  #define MEDIA_BUS_FMT_META_24			0x8007
+#define MEDIA_BUS_FMT_IS_META(code)		\
+	((code) & 0xf000 == 0x7000 || (code) & 0xf000 == 0x8000)
+
  #endif /* __LINUX_MEDIA_BUS_FORMAT_H */

mbus code seems to be u32, so the above won't work. Maybe:

Quoted 5 lines
Quoted 3 lines

(((code) & ~0xfffULL) == 0x7000 || ((code) & ~0xfffULL) == 0x8000)

Also, embedded formats with 0x9nnn codes are added later in the series.

Thanks, I'll address these for v4.

It would be nice to make this an inline function, to avoid evaluating the code twice. I think you can move it to an internal kernel header, it doesn't need to be exposed to userspace.

Quoted 5 lines
Quoted 3 lines

This was meant for user space consumption for setting the buffer type to the video node accordingly. It's certainly possible to do that without this macro, too.

I think userspace would be fine without it :-)

I'll see if this would seem useful in yavta support before dropping it: the buffer type needs to be specified correctly and knowing the format is a metadata format maybe helpful. Let's see.

Signature · 1 line
Sakari Ailus
author maintainer
#40 lore raw

Re: [PATCH v3 06/10] media: uapi: Add a macro to tell whether an mbus code is metadata

Quoted 41 lines · Sakari Ailus
Quoted 40 lines · Laurent Pinchart
Quoted 38 lines · Sakari Ailus
Quoted 26 lines · Laurent Pinchart
Quoted 25 lines · Sakari Ailus
Quoted 24 lines · Tomi Valkeinen
Quoted 12 lines · Sakari Ailus

Add a macro to tell whether a given mbus code is metadata.

---
  include/uapi/linux/media-bus-format.h | 3 +++
  1 file changed, 3 insertions(+)
include/uapi/linux/media-bus-format.h +0 -0
index 9ee031397372..2486b4178c5f 100644

include/uapi/linux/media-bus-format.h line 187

Quoted 6 lines
include/uapi/linux/media-bus-format.h +3 -0
@@ -182,4 +182,7 @@
  #define MEDIA_BUS_FMT_META_20			0x8006
  #define MEDIA_BUS_FMT_META_24			0x8007
+#define MEDIA_BUS_FMT_IS_META(code)		\
+	((code) & 0xf000 == 0x7000 || (code) & 0xf000 == 0x8000)
+
  #endif /* __LINUX_MEDIA_BUS_FORMAT_H */

mbus code seems to be u32, so the above won't work. Maybe:

(((code) & ~0xfffULL) == 0x7000 || ((code) & ~0xfffULL) == 0x8000)

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Also, embedded formats with 0x9nnn codes are added later in the series.

Thanks, I'll address these for v4.

It would be nice to make this an inline function, to avoid evaluating the code twice. I think you can move it to an internal kernel header, it doesn't need to be exposed to userspace.

This was meant for user space consumption for setting the buffer type to the video node accordingly. It's certainly possible to do that without this macro, too.

Quoted 5 lines
Quoted 1 line

I think userspace would be fine without it :-)

I'll see if this would seem useful in yavta support before dropping it: the buffer type needs to be specified correctly and knowing the format is a metadata format maybe helpful. Let's see.

As yavta doesn't deal with mbus codes, this isn't useful for it. I'll drop it for now.

Signature · 1 line
Sakari Ailus
author maintainer
#41 lore raw

PATCH v3 7/10 media: uapi: Add generic 8-bit metadata format definitions

In reply to Sakari Ailus

Generic 8-bit metadata formats define the in-memory data layout but not the format of the data itself. The reasoning for having such formats is to allow CSI-2 receiver drivers to receive and DMA drivers to write the data to memory without knowing a large number of device specific formats.

These formats may be used only in conjunction of a Media controller pipeline where the internal pad of the source sub-device defines the specific format of the data (using an mbus code).

---
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 331 ++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
 include/uapi/linux/videodev2.h                |   9 +
 4 files changed, 349 insertions(+)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
Documentation/userspace-api/media/v4l/meta-formats.rst +1 -0
index 0bb61fc5bc00..919f595576b9 100644
@@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
     metafmt-vsp1-hgo
     metafmt-vsp1-hgt
     metafmt-vivid
+    metafmt-generic
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +331 -0

331 lines not on this page · raw message

drivers/media/v4l2-core/v4l2-ioctl.c +8 -0
index f4d9d6279094..fbbddc333a30 100644
@@ -1452,6 +1452,14 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
+	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8b Generic Meta, 10b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8b Generic Meta, 12b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8b Generic Meta, 14b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8b Generic Meta, 16b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8b Generic Meta, 20b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8b Generic Meta, 24b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_2_24:	descr = "2x8b Generic Meta, 24b CSI-2"; break;
 
 	default:
 		/* Compressed formats */
include/uapi/linux/videodev2.h +9 -0
index 78260e5d9985..b4284a564025 100644
@@ -838,6 +838,15 @@ struct v4l2_pix_format {
 #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
 #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
 
+#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_2_24	v4l2_fourcc('M', 'C', '2', 'O') /* 2 bytes of 8-bit metadata, 24-bit CSI-2 packed */
+
 /* priv field value to indicates that subsequent fields are valid. */
 #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
 
Signature · 1 line
2.39.2
#42 lore raw

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

Quoted 50 lines · Sakari Ailus

Generic 8-bit metadata formats define the in-memory data layout but not the format of the data itself. The reasoning for having such formats is to allow CSI-2 receiver drivers to receive and DMA drivers to write the data to memory without knowing a large number of device specific formats.

These formats may be used only in conjunction of a Media controller pipeline where the internal pad of the source sub-device defines the specific format of the data (using an mbus code).

---
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 331 ++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
 include/uapi/linux/videodev2.h                |   9 +
 4 files changed, 349 insertions(+)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
Documentation/userspace-api/media/v4l/meta-formats.rst +1 -0
index 0bb61fc5bc00..919f595576b9 100644
@@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
     metafmt-vsp1-hgo
     metafmt-vsp1-hgt
     metafmt-vivid
+    metafmt-generic
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +17 -0
new file mode 100644
index 000000000000..a27bfc721edf
@@ -0,0 +1,331 @@
+.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
+
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O')
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+
+
+Generic line-based metadata formats
+
+
+Description
+===========
+
+These generic line-based metadata formats define the memory layout of the data
+without defining the format or meaning of the metadata itself. These formats may
+only be used with a Media controller pipeline where the more specific format is
+defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL>` of the source

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 27

Quoted 10 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +10 -0
+sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`.
+
+.. _v4l2-meta-fmt-generic-8:
+
+V4L2_META_FMT_GENERIC_8
+-----------------------
+
+The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
+
+This format is also used on CSI-2 on both 8 bits per sample as well as on

on both 8 bits per sample as well as on ->
for both 8 bits per sample as well as for

Same elsewhere.

Quoted 29 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +29 -0
+16 bits per sample when two bytes of metadata are packed into one sample.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_8.**
+Each cell is one byte. "M" denotes a byte of metadata.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+    * - start + 4:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+
+.. _v4l2-meta-fmt-generic-csi2-10:
+
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 66

Quoted 10 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +10 -0
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is also used on CSI-2 on 20 bits per sample format that packs two
+bytes of metadata into one sample.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.

I think you should document whether the padding is always 0 or can be any value. Perhaps 'X' is a better 'name' for the padding byte in the latter case.

Quoted 294 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +265 -0
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+    * - start + 5:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-12:
+
+V4L2_META_FMT_GENERIC_CSI2_12
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_12 contains packed 8-bit generic metadata, 12 bits
+for each 8 bits of data. Every four bytes of metadata is followed by two bytes
+of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 6:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-14:
+
+V4L2_META_FMT_GENERIC_CSI2_14
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_14 contains packed 8-bit generic metadata, 14 bits
+for each 8 bits of data. Every four bytes of metadata is followed by three
+bytes of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+      - p
+      - p
+    * - start + 7:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-16:
+
+V4L2_META_FMT_GENERIC_CSI2_16
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_16 contains packed 8-bit generic metadata, 16 bits
+for each 8 bits of data. Every byte of metadata is followed by one byte of
+padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - M\ :sub:`10`
+      - p
+      - M\ :sub:`20`
+      - p
+      - M\ :sub:`30`
+      - p
+    * - start + 8:
+      - M\ :sub:`01`
+      - p
+      - M\ :sub:`11`
+      - p
+      - M\ :sub:`21`
+      - p
+      - M\ :sub:`31`
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-20:
+
+V4L2_META_FMT_GENERIC_CSI2_20
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_20 contains packed 8-bit generic metadata, 20 bits
+for each 8 bits of data. Every byte of metadata is followed by alternating one
+and two bytes of padding. The way the data is stored follows the CSI-2
+specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - M\ :sub:`10`
+      - p
+      - p
+      - M\ :sub:`20`
+      - p
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 10:
+      - M\ :sub:`01`
+      - p
+      - M\ :sub:`11`
+      - p
+      - p
+      - M\ :sub:`21`
+      - p
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-24:
+
+V4L2_META_FMT_GENERIC_CSI2_24
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_24 contains packed 8-bit generic metadata, 24 bits
+for each 8 bits of data. Every byte of metadata is followed by two bytes of
+padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - p
+      - M\ :sub:`10`
+      - p
+      - p
+      - M\ :sub:`20`
+      - p
+      - p
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 12:
+      - M\ :sub:`01`
+      - p
+      - p
+      - M\ :sub:`11`
+      - p
+      - p
+      - M\ :sub:`21`
+      - p
+      - p
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-2-24:
+
+V4L2_META_FMT_GENERIC_CSI2_2_24
+-------------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_2_24 contains packed 8-bit generic metadata, 24 bits
+for each two times 8 bits of data. Every two bytes of metadata are followed by
+one byte of padding. The way the data is stored follows the CSI-2
+specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_2_24.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - p
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+    * - start + 6:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - p
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+
drivers/media/v4l2-core/v4l2-ioctl.c +8 -0
index f4d9d6279094..fbbddc333a30 100644
@@ -1452,6 +1452,14 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
+	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8b Generic Meta, 10b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8b Generic Meta, 12b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8b Generic Meta, 14b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8b Generic Meta, 16b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8b Generic Meta, 20b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8b Generic Meta, 24b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_2_24:	descr = "2x8b Generic Meta, 24b CSI-2"; break;
 
 	default:
 		/* Compressed formats */
include/uapi/linux/videodev2.h +2 -0
index 78260e5d9985..b4284a564025 100644
@@ -838,6 +838,15 @@ struct v4l2_pix_format {
 #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
 #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
 
+#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */

include/uapi/linux/videodev2.h struct v4l2_pix_format { · line 852

Quoted 10 lines
include/uapi/linux/videodev2.h +7 -0
+#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_2_24	v4l2_fourcc('M', 'C', '2', 'O') /* 2 bytes of 8-bit metadata, 24-bit CSI-2 packed */
+
 /* priv field value to indicates that subsequent fields are valid. */
 #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
 

Regards,

Hans

#43 lore raw

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

Hi Sakari, Hans

Quoted 101 lines · Hans Verkuil
Quoted 50 lines · Sakari Ailus

Generic 8-bit metadata formats define the in-memory data layout but not the format of the data itself. The reasoning for having such formats is to allow CSI-2 receiver drivers to receive and DMA drivers to write the data to memory without knowing a large number of device specific formats.

These formats may be used only in conjunction of a Media controller pipeline where the internal pad of the source sub-device defines the specific format of the data (using an mbus code).

---
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 331 ++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
 include/uapi/linux/videodev2.h                |   9 +
 4 files changed, 349 insertions(+)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
Documentation/userspace-api/media/v4l/meta-formats.rst +1 -0
index 0bb61fc5bc00..919f595576b9 100644
@@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
     metafmt-vsp1-hgo
     metafmt-vsp1-hgt
     metafmt-vivid
+    metafmt-generic
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +17 -0
new file mode 100644
index 000000000000..a27bfc721edf
@@ -0,0 +1,331 @@
+.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
+
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O')
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+
+
+Generic line-based metadata formats
+
+
+Description
+===========
+
+These generic line-based metadata formats define the memory layout of the data
+without defining the format or meaning of the metadata itself. These formats may
+only be used with a Media controller pipeline where the more specific format is
+defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL>` of the source

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 27

Quoted 10 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +10 -0
+sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`.
+
+.. _v4l2-meta-fmt-generic-8:
+
+V4L2_META_FMT_GENERIC_8
+-----------------------
+
+The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
+
+This format is also used on CSI-2 on both 8 bits per sample as well as on

on both 8 bits per sample as well as on ->
for both 8 bits per sample as well as for

Same elsewhere.

Quoted 34 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +34 -0
+16 bits per sample when two bytes of metadata are packed into one sample.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_8.**
+Each cell is one byte. "M" denotes a byte of metadata.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+    * - start + 4:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+
+.. _v4l2-meta-fmt-generic-csi2-10:
+
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is also used on CSI-2 on 20 bits per sample format that packs two
+bytes of metadata into one sample.
Quoted 9 lines

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 66

Quoted 5 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +5 -0
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.

I think you should document whether the padding is always 0 or can be any value. Perhaps 'X' is a better 'name' for the padding byte in the latter case.

Did I get this right that this format is supposed to work as the RAW10 CSI-2 packed image format, where 4 bytes contain the higher 8 bits of the 10 bits sample and the 5th byte every 4 contains the lower 2 bits of the previous 4 sample ?

If that's the case, is 'padding' the correct term here ?

Quoted 308 lines
Quoted 294 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +265 -0
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+    * - start + 5:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-12:
+
+V4L2_META_FMT_GENERIC_CSI2_12
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_12 contains packed 8-bit generic metadata, 12 bits
+for each 8 bits of data. Every four bytes of metadata is followed by two bytes
+of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 6:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-14:
+
+V4L2_META_FMT_GENERIC_CSI2_14
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_14 contains packed 8-bit generic metadata, 14 bits
+for each 8 bits of data. Every four bytes of metadata is followed by three
+bytes of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+      - p
+      - p
+    * - start + 7:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-16:
+
+V4L2_META_FMT_GENERIC_CSI2_16
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_16 contains packed 8-bit generic metadata, 16 bits
+for each 8 bits of data. Every byte of metadata is followed by one byte of
+padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - M\ :sub:`10`
+      - p
+      - M\ :sub:`20`
+      - p
+      - M\ :sub:`30`
+      - p
+    * - start + 8:
+      - M\ :sub:`01`
+      - p
+      - M\ :sub:`11`
+      - p
+      - M\ :sub:`21`
+      - p
+      - M\ :sub:`31`
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-20:
+
+V4L2_META_FMT_GENERIC_CSI2_20
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_20 contains packed 8-bit generic metadata, 20 bits
+for each 8 bits of data. Every byte of metadata is followed by alternating one
+and two bytes of padding. The way the data is stored follows the CSI-2
+specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - M\ :sub:`10`
+      - p
+      - p
+      - M\ :sub:`20`
+      - p
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 10:
+      - M\ :sub:`01`
+      - p
+      - M\ :sub:`11`
+      - p
+      - p
+      - M\ :sub:`21`
+      - p
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-24:
+
+V4L2_META_FMT_GENERIC_CSI2_24
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_24 contains packed 8-bit generic metadata, 24 bits
+for each 8 bits of data. Every byte of metadata is followed by two bytes of
+padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - p
+      - M\ :sub:`10`
+      - p
+      - p
+      - M\ :sub:`20`
+      - p
+      - p
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 12:
+      - M\ :sub:`01`
+      - p
+      - p
+      - M\ :sub:`11`
+      - p
+      - p
+      - M\ :sub:`21`
+      - p
+      - p
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-2-24:
+
+V4L2_META_FMT_GENERIC_CSI2_2_24
+-------------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_2_24 contains packed 8-bit generic metadata, 24 bits
+for each two times 8 bits of data. Every two bytes of metadata are followed by
+one byte of padding. The way the data is stored follows the CSI-2
+specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_2_24.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - p
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+    * - start + 6:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - p
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+
drivers/media/v4l2-core/v4l2-ioctl.c +8 -0
index f4d9d6279094..fbbddc333a30 100644
@@ -1452,6 +1452,14 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
+	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8b Generic Meta, 10b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8b Generic Meta, 12b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8b Generic Meta, 14b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8b Generic Meta, 16b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8b Generic Meta, 20b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8b Generic Meta, 24b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_2_24:	descr = "2x8b Generic Meta, 24b CSI-2"; break;
 
 	default:
 		/* Compressed formats */
include/uapi/linux/videodev2.h +2 -0
index 78260e5d9985..b4284a564025 100644
@@ -838,6 +838,15 @@ struct v4l2_pix_format {
 #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
 #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
 
+#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */

include/uapi/linux/videodev2.h struct v4l2_pix_format { · line 852

Quoted 10 lines
include/uapi/linux/videodev2.h +7 -0
+#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_2_24	v4l2_fourcc('M', 'C', '2', 'O') /* 2 bytes of 8-bit metadata, 24-bit CSI-2 packed */
+
 /* priv field value to indicates that subsequent fields are valid. */
 #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
 

Regards,

Hans

author maintainer
#44 lore raw

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

Hi Jacopo,

Quoted 19 lines · Jacopo Mondi
Quoted 9 lines
Quoted 9 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +9 -0
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is also used on CSI-2 on 20 bits per sample format that packs two
+bytes of metadata into one sample.
Quoted 9 lines

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 66

Quoted 5 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +5 -0
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.

I think you should document whether the padding is always 0 or can be any value. Perhaps 'X' is a better 'name' for the padding byte in the latter case.

Quoted 6 lines

Did I get this right that this format is supposed to work as the RAW10 CSI-2 packed image format, where 4 bytes contain the higher 8 bits of the 10 bits sample and the 5th byte every 4 contains the lower 2 bits of the previous 4 sample ?

If that's the case, is 'padding' the correct term here ?

What else would you call it? It'll be zeros that exist just due to the bit depth used and as such not interesting at all.

Signature · 3 lines
Regards,

Sakari Ailus
#45 lore raw

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

Hi Sakari

Quoted 27 lines · Sakari Ailus

Hi Jacopo,

Quoted 24 lines · Jacopo Mondi
Quoted 9 lines
Quoted 9 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +9 -0
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is also used on CSI-2 on 20 bits per sample format that packs two
+bytes of metadata into one sample.
Quoted 9 lines

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 66

Quoted 5 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +5 -0
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.

I think you should document whether the padding is always 0 or can be any value. Perhaps 'X' is a better 'name' for the padding byte in the latter case.

Did I get this right that this format is supposed to work as the RAW10 CSI-2 packed image format, where 4 bytes contain the higher 8 bits of the 10 bits sample and the 5th byte every 4 contains the lower 2 bits of the previous 4 sample ?

Quoted 4 lines
Quoted 1 line

If that's the case, is 'padding' the correct term here ?

What else would you call it? It'll be zeros that exist just due to the bit depth used and as such not interesting at all.

Ah, they will be 0s! They won't contain the 2 lower bits of the previous 4 bytes then ?

I guess this is due to the fact the number of valid bits in each metadata sample is always 8, regardless of the sample size on the media bus ?

The DMA engine will re-pack them to CSI2_10 (or CSI2_12 etc) just to match the image format sizes then ?

In this case, padding is indeed correct, sorry for the noise!

Thanks
j

Quoted 5 lines

--
Regards,

Sakari Ailus

author maintainer
#46 lore raw

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

Hi Jacopo,

Quoted 35 lines · Jacopo Mondi

Hi Sakari

Quoted 27 lines · Sakari Ailus

Hi Jacopo,

Quoted 24 lines · Jacopo Mondi
Quoted 9 lines
Quoted 9 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +9 -0
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is also used on CSI-2 on 20 bits per sample format that packs two
+bytes of metadata into one sample.
Quoted 9 lines

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 66

Quoted 5 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +5 -0
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.

I think you should document whether the padding is always 0 or can be any value. Perhaps 'X' is a better 'name' for the padding byte in the latter case.

Did I get this right that this format is supposed to work as the RAW10 CSI-2 packed image format, where 4 bytes contain the higher 8 bits of the 10 bits sample and the 5th byte every 4 contains the lower 2 bits of the previous 4 sample ?

Quoted 4 lines
Quoted 1 line

If that's the case, is 'padding' the correct term here ?

What else would you call it? It'll be zeros that exist just due to the bit depth used and as such not interesting at all.

Quoted 6 lines

Ah, they will be 0s! They won't contain the 2 lower bits of the previous 4 bytes then ?

I guess this is due to the fact the number of valid bits in each metadata sample is always 8, regardless of the sample size on the media bus ?

Correct. This is also documented for the generic metadata formats.

Quoted 3 lines

The DMA engine will re-pack them to CSI2_10 (or CSI2_12 etc) just to match the image format sizes then ?

Generally yes. The receiver hardware / DMA is of course free to strip out the padding but in practice that doesn't seem to be done.

Quoted 2 lines

In this case, padding is indeed correct, sorry for the noise!

No problem.

Signature · 3 lines
Regards,

Sakari Ailus
maintainer
#47 lore raw

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

In reply to Sakari Ailus

Hi Sakari,

Quoted 24 lines · Sakari Ailus
Quoted 23 lines · Jacopo Mondi
Quoted 9 lines
Quoted 9 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +9 -0
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is also used on CSI-2 on 20 bits per sample format that packs two
+bytes of metadata into one sample.
Quoted 8 lines

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 66

Quoted 5 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +5 -0
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.

I think you should document whether the padding is always 0 or can be any value. Perhaps 'X' is a better 'name' for the padding byte in the latter case.

Did I get this right that this format is supposed to work as the RAW10 CSI-2 packed image format, where 4 bytes contain the higher 8 bits of the 10 bits sample and the 5th byte every 4 contains the lower 2 bits of the previous 4 sample ?

Quoted 4 lines
Quoted 1 line

If that's the case, is 'padding' the correct term here ?

What else would you call it? It'll be zeros that exist just due to the bit depth used and as such not interesting at all.

It's actually not 0, CCS requires the padding bytes to be 0x55.

I wonder if the conformance test suite tests the contents of the padding bytes.

Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#48 lore raw

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

Hi Laurent,

Quoted 29 lines · Laurent Pinchart

Hi Sakari,

Quoted 26 lines · Sakari Ailus
Quoted 24 lines · Jacopo Mondi
Quoted 9 lines
Quoted 9 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +9 -0
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is also used on CSI-2 on 20 bits per sample format that packs two
+bytes of metadata into one sample.
Quoted 8 lines

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 66

Quoted 5 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +5 -0
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.

I think you should document whether the padding is always 0 or can be any value. Perhaps 'X' is a better 'name' for the padding byte in the latter case.

Did I get this right that this format is supposed to work as the RAW10 CSI-2 packed image format, where 4 bytes contain the higher 8 bits of the 10 bits sample and the 5th byte every 4 contains the lower 2 bits of the previous 4 sample ?

If that's the case, is 'padding' the correct term here ?

Quoted 7 lines
Quoted 2 lines

What else would you call it? It'll be zeros that exist just due to the bit depth used and as such not interesting at all.

It's actually not 0, CCS requires the padding bytes to be 0x55.

I wonder if the conformance test suite tests the contents of the padding bytes.

I don't know. I could add the value is unspecified but as it has not been specified, there's no change in meaning (just size).

Signature · 3 lines
Regards,

Sakari Ailus
maintainer
#49 lore raw

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

CC'ing Naush and David

Quoted 31 lines · Sakari Ailus
Quoted 30 lines · Laurent Pinchart
Quoted 28 lines · Sakari Ailus
Quoted 18 lines · Jacopo Mondi
Quoted 9 lines
Quoted 9 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +9 -0
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is also used on CSI-2 on 20 bits per sample format that packs two
+bytes of metadata into one sample.
Quoted 8 lines

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 66

Quoted 5 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +5 -0
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.

I think you should document whether the padding is always 0 or can be any value. Perhaps 'X' is a better 'name' for the padding byte in the latter case.

Quoted 6 lines

Did I get this right that this format is supposed to work as the RAW10 CSI-2 packed image format, where 4 bytes contain the higher 8 bits of the 10 bits sample and the 5th byte every 4 contains the lower 2 bits of the previous 4 sample ?

If that's the case, is 'padding' the correct term here ?

What else would you call it? It'll be zeros that exist just due to the bit depth used and as such not interesting at all.

Quoted 7 lines
Quoted 4 lines

It's actually not 0, CCS requires the padding bytes to be 0x55.

I wonder if the conformance test suite tests the contents of the padding bytes.

I don't know. I could add the value is unspecified but as it has not been specified, there's no change in meaning (just size).

I started writing that I don't see how it could help applications to know that the padding byte is 0x55, but the SMIA++ embedded data parser in libcamera actually checks for it, and considers the embedded data to be erroneous if it has a different value.

Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#50 lore raw

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

Hi Laurent,

Quoted 39 lines · Laurent Pinchart

CC'ing Naush and David

Quoted 36 lines · Sakari Ailus
Quoted 34 lines · Laurent Pinchart
Quoted 24 lines · Sakari Ailus
Quoted 23 lines · Jacopo Mondi
Quoted 9 lines
Quoted 9 lines
(quoted diff) +9 -0
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is also used on CSI-2 on 20 bits per sample format that packs two
+bytes of metadata into one sample.
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.

I think you should document whether the padding is always 0 or can be any value. Perhaps 'X' is a better 'name' for the padding byte in the latter case.

Did I get this right that this format is supposed to work as the RAW10 CSI-2 packed image format, where 4 bytes contain the higher 8 bits of the 10 bits sample and the 5th byte every 4 contains the lower 2 bits of the previous 4 sample ?

Quoted 4 lines
Quoted 1 line

If that's the case, is 'padding' the correct term here ?

What else would you call it? It'll be zeros that exist just due to the bit depth used and as such not interesting at all.

It's actually not 0, CCS requires the padding bytes to be 0x55.

I wonder if the conformance test suite tests the contents of the padding bytes.

Quoted 7 lines
Quoted 2 lines

I don't know. I could add the value is unspecified but as it has not been specified, there's no change in meaning (just size).

I started writing that I don't see how it could help applications to know that the padding byte is 0x55, but the SMIA++ embedded data parser in libcamera actually checks for it, and considers the embedded data to be erroneous if it has a different value.

I think it's fine to check for it if you know it's CCS/SMIA++/SMIA embedded data. But documenting it here isn't a great idea as then other uses of this format definition would be excluded.

Signature · 3 lines
Regards,

Sakari Ailus
maintainer
#51 lore raw

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

Quoted 41 lines · Sakari Ailus
Quoted 40 lines · Laurent Pinchart
Quoted 38 lines · Sakari Ailus
Quoted 27 lines · Laurent Pinchart
Quoted 26 lines · Sakari Ailus
Quoted 24 lines · Jacopo Mondi
Quoted 9 lines
Quoted 9 lines
(quoted diff) +9 -0
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is also used on CSI-2 on 20 bits per sample format that packs two
+bytes of metadata into one sample.
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.

I think you should document whether the padding is always 0 or can be any value. Perhaps 'X' is a better 'name' for the padding byte in the latter case.

Did I get this right that this format is supposed to work as the RAW10 CSI-2 packed image format, where 4 bytes contain the higher 8 bits of the 10 bits sample and the 5th byte every 4 contains the lower 2 bits of the previous 4 sample ?

If that's the case, is 'padding' the correct term here ?

Quoted 7 lines
Quoted 2 lines

What else would you call it? It'll be zeros that exist just due to the bit depth used and as such not interesting at all.

It's actually not 0, CCS requires the padding bytes to be 0x55.

I wonder if the conformance test suite tests the contents of the padding bytes.

I don't know. I could add the value is unspecified but as it has not been specified, there's no change in meaning (just size).

Quoted 8 lines
Quoted 4 lines

I started writing that I don't see how it could help applications to know that the padding byte is 0x55, but the SMIA++ embedded data parser in libcamera actually checks for it, and considers the embedded data to be erroneous if it has a different value.

I think it's fine to check for it if you know it's CCS/SMIA++/SMIA embedded data. But documenting it here isn't a great idea as then other uses of this format definition would be excluded.

I'm OK with that, but note that you've mentioned in a different patch in the same series that you wouldn't use the CCS media bus code for sensors that are compliant with the CCS packing and encoding but not the CCS register set. That's not very consistent :-)

Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#52 lore raw

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

Hi Laurent,

Quoted 47 lines · Laurent Pinchart
Quoted 46 lines · Sakari Ailus
Quoted 44 lines · Laurent Pinchart
Quoted 31 lines · Sakari Ailus
Quoted 30 lines · Laurent Pinchart
Quoted 28 lines · Sakari Ailus
Quoted 18 lines · Jacopo Mondi
Quoted 9 lines
Quoted 9 lines
(quoted diff) +9 -0
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is also used on CSI-2 on 20 bits per sample format that packs two
+bytes of metadata into one sample.
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.

I think you should document whether the padding is always 0 or can be any value. Perhaps 'X' is a better 'name' for the padding byte in the latter case.

Quoted 6 lines

Did I get this right that this format is supposed to work as the RAW10 CSI-2 packed image format, where 4 bytes contain the higher 8 bits of the 10 bits sample and the 5th byte every 4 contains the lower 2 bits of the previous 4 sample ?

If that's the case, is 'padding' the correct term here ?

What else would you call it? It'll be zeros that exist just due to the bit depth used and as such not interesting at all.

Quoted 7 lines
Quoted 4 lines

It's actually not 0, CCS requires the padding bytes to be 0x55.

I wonder if the conformance test suite tests the contents of the padding bytes.

I don't know. I could add the value is unspecified but as it has not been specified, there's no change in meaning (just size).

I started writing that I don't see how it could help applications to know that the padding byte is 0x55, but the SMIA++ embedded data parser in libcamera actually checks for it, and considers the embedded data to be erroneous if it has a different value.

Quoted 8 lines
Quoted 3 lines

I think it's fine to check for it if you know it's CCS/SMIA++/SMIA embedded data. But documenting it here isn't a great idea as then other uses of this format definition would be excluded.

I'm OK with that, but note that you've mentioned in a different patch in the same series that you wouldn't use the CCS media bus code for sensors that are compliant with the CCS packing and encoding but not the CCS register set. That's not very consistent :-)

Which specific patch are you referring to?

Signature · 3 lines
Regards,

Sakari Ailus
maintainer
#53 lore raw

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

Quoted 54 lines · Sakari Ailus

Hi Laurent,

Quoted 51 lines · Laurent Pinchart
Quoted 49 lines · Sakari Ailus
Quoted 37 lines · Laurent Pinchart
Quoted 36 lines · Sakari Ailus
Quoted 34 lines · Laurent Pinchart
Quoted 24 lines · Sakari Ailus
Quoted 23 lines · Jacopo Mondi
Quoted 9 lines
Quoted 9 lines
(quoted diff) +9 -0
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is also used on CSI-2 on 20 bits per sample format that packs two
+bytes of metadata into one sample.
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.

I think you should document whether the padding is always 0 or can be any value. Perhaps 'X' is a better 'name' for the padding byte in the latter case.

Did I get this right that this format is supposed to work as the RAW10 CSI-2 packed image format, where 4 bytes contain the higher 8 bits of the 10 bits sample and the 5th byte every 4 contains the lower 2 bits of the previous 4 sample ?

Quoted 4 lines
Quoted 1 line

If that's the case, is 'padding' the correct term here ?

What else would you call it? It'll be zeros that exist just due to the bit depth used and as such not interesting at all.

It's actually not 0, CCS requires the padding bytes to be 0x55.

I wonder if the conformance test suite tests the contents of the padding bytes.

Quoted 7 lines
Quoted 2 lines

I don't know. I could add the value is unspecified but as it has not been specified, there's no change in meaning (just size).

I started writing that I don't see how it could help applications to know that the padding byte is 0x55, but the SMIA++ embedded data parser in libcamera actually checks for it, and considers the embedded data to be erroneous if it has a different value.

I think it's fine to check for it if you know it's CCS/SMIA++/SMIA embedded data. But documenting it here isn't a great idea as then other uses of this format definition would be excluded.

Quoted 6 lines
Quoted 4 lines

I'm OK with that, but note that you've mentioned in a different patch in the same series that you wouldn't use the CCS media bus code for sensors that are compliant with the CCS packing and encoding but not the CCS register set. That's not very consistent :-)

Which specific patch are you referring to?

"[PATCH v3 09/10] media: Add media bus codes for MIPI CCS embedded data"

You wrote

"There are sensors that aren't fully compatible with CCS (including those compatible with SMIA and SMIA++) but I wouldn't expect the format to be used by devices that are entirely incompatible with CCS."

Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#54 lore raw

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

Hi Laurent,

Quoted 62 lines · Laurent Pinchart
Quoted 54 lines · Sakari Ailus

Hi Laurent,

Quoted 51 lines · Laurent Pinchart
Quoted 49 lines · Sakari Ailus
Quoted 37 lines · Laurent Pinchart
Quoted 36 lines · Sakari Ailus
Quoted 34 lines · Laurent Pinchart
Quoted 24 lines · Sakari Ailus
Quoted 23 lines · Jacopo Mondi
Quoted 9 lines
Quoted 9 lines
(quoted diff) +9 -0
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is also used on CSI-2 on 20 bits per sample format that packs two
+bytes of metadata into one sample.
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.

I think you should document whether the padding is always 0 or can be any value. Perhaps 'X' is a better 'name' for the padding byte in the latter case.

Did I get this right that this format is supposed to work as the RAW10 CSI-2 packed image format, where 4 bytes contain the higher 8 bits of the 10 bits sample and the 5th byte every 4 contains the lower 2 bits of the previous 4 sample ?

Quoted 4 lines
Quoted 1 line

If that's the case, is 'padding' the correct term here ?

What else would you call it? It'll be zeros that exist just due to the bit depth used and as such not interesting at all.

It's actually not 0, CCS requires the padding bytes to be 0x55.

I wonder if the conformance test suite tests the contents of the padding bytes.

Quoted 7 lines
Quoted 2 lines

I don't know. I could add the value is unspecified but as it has not been specified, there's no change in meaning (just size).

I started writing that I don't see how it could help applications to know that the padding byte is 0x55, but the SMIA++ embedded data parser in libcamera actually checks for it, and considers the embedded data to be erroneous if it has a different value.

I think it's fine to check for it if you know it's CCS/SMIA++/SMIA embedded data. But documenting it here isn't a great idea as then other uses of this format definition would be excluded.

Quoted 6 lines
Quoted 4 lines

I'm OK with that, but note that you've mentioned in a different patch in the same series that you wouldn't use the CCS media bus code for sensors that are compliant with the CCS packing and encoding but not the CCS register set. That's not very consistent :-)

Which specific patch are you referring to?

Quoted 7 lines

"[PATCH v3 09/10] media: Add media bus codes for MIPI CCS embedded data"

You wrote

"There are sensors that aren't fully compatible with CCS (including those compatible with SMIA and SMIA++) but I wouldn't expect the format to be used by devices that are entirely incompatible with CCS."

Ah, right.

I meant that if a sensor implementation isn't related to any of these standards, it's highly unlikely to use the same embedded data format. Of course, if it does, then I think we should use the mbus code, too. But I'm not holding my breath.

Signature · 3 lines
Regards,

Sakari Ailus
author maintainer
#55 lore raw

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

Quoted 73 lines · Sakari Ailus

Hi Laurent,

Quoted 62 lines · Laurent Pinchart
Quoted 54 lines · Sakari Ailus

Hi Laurent,

Quoted 51 lines · Laurent Pinchart
Quoted 49 lines · Sakari Ailus
Quoted 37 lines · Laurent Pinchart
Quoted 36 lines · Sakari Ailus
Quoted 34 lines · Laurent Pinchart
Quoted 24 lines · Sakari Ailus
Quoted 23 lines · Jacopo Mondi
Quoted 9 lines
Quoted 9 lines
(quoted diff) +9 -0
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is also used on CSI-2 on 20 bits per sample format that packs two
+bytes of metadata into one sample.
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.

I think you should document whether the padding is always 0 or can be any value. Perhaps 'X' is a better 'name' for the padding byte in the latter case.

Did I get this right that this format is supposed to work as the RAW10 CSI-2 packed image format, where 4 bytes contain the higher 8 bits of the 10 bits sample and the 5th byte every 4 contains the lower 2 bits of the previous 4 sample ?

Quoted 4 lines
Quoted 1 line

If that's the case, is 'padding' the correct term here ?

What else would you call it? It'll be zeros that exist just due to the bit depth used and as such not interesting at all.

It's actually not 0, CCS requires the padding bytes to be 0x55.

I wonder if the conformance test suite tests the contents of the padding bytes.

Quoted 7 lines
Quoted 2 lines

I don't know. I could add the value is unspecified but as it has not been specified, there's no change in meaning (just size).

I started writing that I don't see how it could help applications to know that the padding byte is 0x55, but the SMIA++ embedded data parser in libcamera actually checks for it, and considers the embedded data to be erroneous if it has a different value.

I think it's fine to check for it if you know it's CCS/SMIA++/SMIA embedded data. But documenting it here isn't a great idea as then other uses of this format definition would be excluded.

Quoted 6 lines
Quoted 4 lines

I'm OK with that, but note that you've mentioned in a different patch in the same series that you wouldn't use the CCS media bus code for sensors that are compliant with the CCS packing and encoding but not the CCS register set. That's not very consistent :-)

Which specific patch are you referring to?

Quoted 7 lines

"[PATCH v3 09/10] media: Add media bus codes for MIPI CCS embedded data"

You wrote

"There are sensors that aren't fully compatible with CCS (including those compatible with SMIA and SMIA++) but I wouldn't expect the format to be used by devices that are entirely incompatible with CCS."

Quoted 6 lines

Ah, right.

I meant that if a sensor implementation isn't related to any of these standards, it's highly unlikely to use the same embedded data format. Of course, if it does, then I think we should use the mbus code, too. But I'm not holding my breath.

To clarify: to use that format, I'd expect the sensor to use standard CCS registers. If it doesn't, a new mbus code should be added and documented. The CCS embedded data header isn't enough to justify the use of the mbus code --- the documentation here also says "which is used to store the register configuration used for capturing a given frame".

Signature · 3 lines
Regards,

Sakari Ailus
author maintainer
#56 lore raw

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

In reply to Hans Verkuil

Hi Hans,

Thanks for the review.

Quoted 65 lines · Hans Verkuil
Quoted 50 lines · Sakari Ailus

Generic 8-bit metadata formats define the in-memory data layout but not the format of the data itself. The reasoning for having such formats is to allow CSI-2 receiver drivers to receive and DMA drivers to write the data to memory without knowing a large number of device specific formats.

These formats may be used only in conjunction of a Media controller pipeline where the internal pad of the source sub-device defines the specific format of the data (using an mbus code).

---
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 331 ++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
 include/uapi/linux/videodev2.h                |   9 +
 4 files changed, 349 insertions(+)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
Documentation/userspace-api/media/v4l/meta-formats.rst +1 -0
index 0bb61fc5bc00..919f595576b9 100644
@@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
     metafmt-vsp1-hgo
     metafmt-vsp1-hgt
     metafmt-vivid
+    metafmt-generic
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +17 -0
new file mode 100644
index 000000000000..a27bfc721edf
@@ -0,0 +1,331 @@
+.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
+
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O')
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+
+
+Generic line-based metadata formats
+
+
+Description
+===========
+
+These generic line-based metadata formats define the memory layout of the data
+without defining the format or meaning of the metadata itself. These formats may
+only be used with a Media controller pipeline where the more specific format is
+defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL>` of the source

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 27

Quoted 10 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +10 -0
+sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`.
+
+.. _v4l2-meta-fmt-generic-8:
+
+V4L2_META_FMT_GENERIC_8
+-----------------------
+
+The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
+
+This format is also used on CSI-2 on both 8 bits per sample as well as on

on both 8 bits per sample as well as on ->
for both 8 bits per sample as well as for

Quoted 1 line

Same elsewhere.

There's another for 10 bits per sample, the usage is different elsewhere as there is only a single bit depth in other cases.

Quoted 35 lines
Quoted 35 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +34 -0
+16 bits per sample when two bytes of metadata are packed into one sample.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_8.**
+Each cell is one byte. "M" denotes a byte of metadata.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+    * - start + 4:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+
+.. _v4l2-meta-fmt-generic-csi2-10:
+
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is also used on CSI-2 on 20 bits per sample format that packs two
+bytes of metadata into one sample.
Quoted 8 lines

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 66

Quoted 5 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +5 -0
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.

I think you should document whether the padding is always 0 or can be any value. Perhaps 'X' is a better 'name' for the padding byte in the latter case.

I'll change these for v4.

Quoted 309 lines
Quoted 295 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +265 -0
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+    * - start + 5:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-12:
+
+V4L2_META_FMT_GENERIC_CSI2_12
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_12 contains packed 8-bit generic metadata, 12 bits
+for each 8 bits of data. Every four bytes of metadata is followed by two bytes
+of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 6:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-14:
+
+V4L2_META_FMT_GENERIC_CSI2_14
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_14 contains packed 8-bit generic metadata, 14 bits
+for each 8 bits of data. Every four bytes of metadata is followed by three
+bytes of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+      - p
+      - p
+    * - start + 7:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-16:
+
+V4L2_META_FMT_GENERIC_CSI2_16
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_16 contains packed 8-bit generic metadata, 16 bits
+for each 8 bits of data. Every byte of metadata is followed by one byte of
+padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - M\ :sub:`10`
+      - p
+      - M\ :sub:`20`
+      - p
+      - M\ :sub:`30`
+      - p
+    * - start + 8:
+      - M\ :sub:`01`
+      - p
+      - M\ :sub:`11`
+      - p
+      - M\ :sub:`21`
+      - p
+      - M\ :sub:`31`
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-20:
+
+V4L2_META_FMT_GENERIC_CSI2_20
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_20 contains packed 8-bit generic metadata, 20 bits
+for each 8 bits of data. Every byte of metadata is followed by alternating one
+and two bytes of padding. The way the data is stored follows the CSI-2
+specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - M\ :sub:`10`
+      - p
+      - p
+      - M\ :sub:`20`
+      - p
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 10:
+      - M\ :sub:`01`
+      - p
+      - M\ :sub:`11`
+      - p
+      - p
+      - M\ :sub:`21`
+      - p
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-24:
+
+V4L2_META_FMT_GENERIC_CSI2_24
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_24 contains packed 8-bit generic metadata, 24 bits
+for each 8 bits of data. Every byte of metadata is followed by two bytes of
+padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - p
+      - M\ :sub:`10`
+      - p
+      - p
+      - M\ :sub:`20`
+      - p
+      - p
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 12:
+      - M\ :sub:`01`
+      - p
+      - p
+      - M\ :sub:`11`
+      - p
+      - p
+      - M\ :sub:`21`
+      - p
+      - p
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-2-24:
+
+V4L2_META_FMT_GENERIC_CSI2_2_24
+-------------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_2_24 contains packed 8-bit generic metadata, 24 bits
+for each two times 8 bits of data. Every two bytes of metadata are followed by
+one byte of padding. The way the data is stored follows the CSI-2
+specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_2_24.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - p
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+    * - start + 6:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - p
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+
drivers/media/v4l2-core/v4l2-ioctl.c +8 -0
index f4d9d6279094..fbbddc333a30 100644
@@ -1452,6 +1452,14 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
+	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8b Generic Meta, 10b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8b Generic Meta, 12b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8b Generic Meta, 14b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8b Generic Meta, 16b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8b Generic Meta, 20b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8b Generic Meta, 24b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_2_24:	descr = "2x8b Generic Meta, 24b CSI-2"; break;
 
 	default:
 		/* Compressed formats */
include/uapi/linux/videodev2.h +2 -0
index 78260e5d9985..b4284a564025 100644
@@ -838,6 +838,15 @@ struct v4l2_pix_format {
 #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
 #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
 
+#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */

include/uapi/linux/videodev2.h struct v4l2_pix_format { · line 852

Quoted 10 lines
include/uapi/linux/videodev2.h +7 -0
+#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_2_24	v4l2_fourcc('M', 'C', '2', 'O') /* 2 bytes of 8-bit metadata, 24-bit CSI-2 packed */
+
 /* priv field value to indicates that subsequent fields are valid. */
 #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
 

Regards,

Hans

Signature · 3 lines
Regards,

Sakari Ailus
maintainer
#57 lore raw

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

In reply to Sakari Ailus
Quoted 50 lines · Sakari Ailus

Generic 8-bit metadata formats define the in-memory data layout but not the format of the data itself. The reasoning for having such formats is to allow CSI-2 receiver drivers to receive and DMA drivers to write the data to memory without knowing a large number of device specific formats.

These formats may be used only in conjunction of a Media controller pipeline where the internal pad of the source sub-device defines the specific format of the data (using an mbus code).

---
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 331 ++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
 include/uapi/linux/videodev2.h                |   9 +
 4 files changed, 349 insertions(+)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
Documentation/userspace-api/media/v4l/meta-formats.rst +1 -0
index 0bb61fc5bc00..919f595576b9 100644
@@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
     metafmt-vsp1-hgo
     metafmt-vsp1-hgt
     metafmt-vivid
+    metafmt-generic
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +17 -0
new file mode 100644
index 000000000000..a27bfc721edf
@@ -0,0 +1,331 @@
+.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
+
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O')
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+
+
+Generic line-based metadata formats
+
+
+Description
+===========
+
+These generic line-based metadata formats define the memory layout of the data
+without defining the format or meaning of the metadata itself. These formats may
+only be used with a Media controller pipeline where the more specific format is
+defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL>` of the source

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 27

Quoted 10 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +10 -0
+sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`.
+
+.. _v4l2-meta-fmt-generic-8:
+
+V4L2_META_FMT_GENERIC_8
+-----------------------
+
+The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
+
+This format is also used on CSI-2 on both 8 bits per sample as well as on

s/also on/by/

I would also mention "MIPI CCS" instead of "CSI-2".

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 28

Quoted 1 line
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +1 -0
+16 bits per sample when two bytes of metadata are packed into one sample.

"bits per sample" is very ill-defined for metadata, as there's no sample. I would write "for both the RAW8 packing and the 2 bytes RAW16 packing" or something similar.

Similar comments for below.

Quoted 20 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +20 -0
+
+**Byte Order Of V4L2_META_FMT_GENERIC_8.**
+Each cell is one byte. "M" denotes a byte of metadata.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+    * - start + 4:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`

Documentation/userspace-api/media/v4l/metafmt-generic.rst

Quoted 10 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +10 -0
+      - M\ :sub:`31`
+
+.. _v4l2-meta-fmt-generic-csi2-10:
+
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding.

It sounds really weird to write that this format writes 10 bits for each 8 bits of data, when essentially it adds a packing byte every four bytes.

Quoted 1 line

The way the data is stored follows the CSI-2 specification.

Again, it's CCS, not CSI-2.

Quoted 312 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +273 -0
+
+This format is also used on CSI-2 on 20 bits per sample format that packs two
+bytes of metadata into one sample.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+    * - start + 5:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-12:
+
+V4L2_META_FMT_GENERIC_CSI2_12
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_12 contains packed 8-bit generic metadata, 12 bits
+for each 8 bits of data. Every four bytes of metadata is followed by two bytes
+of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 6:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-14:
+
+V4L2_META_FMT_GENERIC_CSI2_14
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_14 contains packed 8-bit generic metadata, 14 bits
+for each 8 bits of data. Every four bytes of metadata is followed by three
+bytes of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+      - p
+      - p
+    * - start + 7:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-16:
+
+V4L2_META_FMT_GENERIC_CSI2_16
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_16 contains packed 8-bit generic metadata, 16 bits
+for each 8 bits of data. Every byte of metadata is followed by one byte of
+padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - M\ :sub:`10`
+      - p
+      - M\ :sub:`20`
+      - p
+      - M\ :sub:`30`
+      - p
+    * - start + 8:
+      - M\ :sub:`01`
+      - p
+      - M\ :sub:`11`
+      - p
+      - M\ :sub:`21`
+      - p
+      - M\ :sub:`31`
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-20:
+
+V4L2_META_FMT_GENERIC_CSI2_20
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_20 contains packed 8-bit generic metadata, 20 bits
+for each 8 bits of data. Every byte of metadata is followed by alternating one
+and two bytes of padding. The way the data is stored follows the CSI-2
+specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - M\ :sub:`10`
+      - p
+      - p
+      - M\ :sub:`20`
+      - p
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 10:
+      - M\ :sub:`01`
+      - p
+      - M\ :sub:`11`
+      - p
+      - p
+      - M\ :sub:`21`
+      - p
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-24:
+
+V4L2_META_FMT_GENERIC_CSI2_24
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_24 contains packed 8-bit generic metadata, 24 bits
+for each 8 bits of data. Every byte of metadata is followed by two bytes of
+padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - p
+      - M\ :sub:`10`
+      - p
+      - p
+      - M\ :sub:`20`
+      - p
+      - p
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 12:
+      - M\ :sub:`01`
+      - p
+      - p
+      - M\ :sub:`11`
+      - p
+      - p
+      - M\ :sub:`21`
+      - p
+      - p
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-2-24:
+
+V4L2_META_FMT_GENERIC_CSI2_2_24
+-------------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_2_24 contains packed 8-bit generic metadata, 24 bits
+for each two times 8 bits of data. Every two bytes of metadata are followed by
+one byte of padding. The way the data is stored follows the CSI-2
+specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_2_24.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - p
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+    * - start + 6:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - p
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+
drivers/media/v4l2-core/v4l2-ioctl.c +8 -0
index f4d9d6279094..fbbddc333a30 100644
@@ -1452,6 +1452,14 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
+	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8b Generic Meta, 10b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8b Generic Meta, 12b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8b Generic Meta, 14b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8b Generic Meta, 16b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8b Generic Meta, 20b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8b Generic Meta, 24b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_2_24:	descr = "2x8b Generic Meta, 24b CSI-2"; break;
 
 	default:
 		/* Compressed formats */
include/uapi/linux/videodev2.h +9 -0
index 78260e5d9985..b4284a564025 100644
@@ -838,6 +838,15 @@ struct v4l2_pix_format {
 #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
 #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
 
+#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_2_24	v4l2_fourcc('M', 'C', '2', 'O') /* 2 bytes of 8-bit metadata, 24-bit CSI-2 packed */
+
 /* priv field value to indicates that subsequent fields are valid. */
 #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
 
Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#58 lore raw

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

Hi Laurent,

Quoted 64 lines · Laurent Pinchart
Quoted 50 lines · Sakari Ailus

Generic 8-bit metadata formats define the in-memory data layout but not the format of the data itself. The reasoning for having such formats is to allow CSI-2 receiver drivers to receive and DMA drivers to write the data to memory without knowing a large number of device specific formats.

These formats may be used only in conjunction of a Media controller pipeline where the internal pad of the source sub-device defines the specific format of the data (using an mbus code).

---
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 331 ++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
 include/uapi/linux/videodev2.h                |   9 +
 4 files changed, 349 insertions(+)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
Documentation/userspace-api/media/v4l/meta-formats.rst +1 -0
index 0bb61fc5bc00..919f595576b9 100644
@@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
     metafmt-vsp1-hgo
     metafmt-vsp1-hgt
     metafmt-vivid
+    metafmt-generic
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +17 -0
new file mode 100644
index 000000000000..a27bfc721edf
@@ -0,0 +1,331 @@
+.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
+
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O')
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+
+
+Generic line-based metadata formats
+
+
+Description
+===========
+
+These generic line-based metadata formats define the memory layout of the data
+without defining the format or meaning of the metadata itself. These formats may
+only be used with a Media controller pipeline where the more specific format is
+defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL>` of the source

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 27

Quoted 10 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +10 -0
+sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`.
+
+.. _v4l2-meta-fmt-generic-8:
+
+V4L2_META_FMT_GENERIC_8
+-----------------------
+
+The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
+
+This format is also used on CSI-2 on both 8 bits per sample as well as on

s/also on/by/

Quoted 1 line

I would also mention "MIPI CCS" instead of "CSI-2".

If CCS were to be mentioned here, then all uses of this format should be included as well.

Quoted 8 lines

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 28

Quoted 2 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +1 -0
+16 bits per sample when two bytes of metadata are packed into one sample.

"bits per sample" is very ill-defined for metadata, as there's no sample. I would write "for both the RAW8 packing and the 2 bytes RAW16 packing" or something similar.

Similar comments for below.

From CSI-2 bus point of view there's no difference between pixel and embedded data when it comes to encoding that data. "Sample" is the next best term beyond "pixel", as the bus can carry samples that may or may not be pixel data. But I'm fine with changing the wording if you think it makes it more understandable.

Quoted 27 lines
Quoted 27 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +26 -0
+
+**Byte Order Of V4L2_META_FMT_GENERIC_8.**
+Each cell is one byte. "M" denotes a byte of metadata.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+    * - start + 4:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+
+.. _v4l2-meta-fmt-generic-csi2-10:
+
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
Quoted 8 lines

Documentation/userspace-api/media/v4l/metafmt-generic.rst

Quoted 4 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +4 -0
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding.

It sounds really weird to write that this format writes 10 bits for each 8 bits of data, when essentially it adds a packing byte every four bytes.

That's how the hardware has been implemented and probably there is a hardware implementation related reason for this.

Quoted 4 lines
Quoted 2 lines

The way the data is stored follows the CSI-2 specification.

Again, it's CCS, not CSI-2.

The packing comes from CSI-2 but CCS adds padding. I'll add this information.

Quoted 314 lines
Quoted 313 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +273 -0
+
+This format is also used on CSI-2 on 20 bits per sample format that packs two
+bytes of metadata into one sample.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+    * - start + 5:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-12:
+
+V4L2_META_FMT_GENERIC_CSI2_12
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_12 contains packed 8-bit generic metadata, 12 bits
+for each 8 bits of data. Every four bytes of metadata is followed by two bytes
+of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 6:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-14:
+
+V4L2_META_FMT_GENERIC_CSI2_14
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_14 contains packed 8-bit generic metadata, 14 bits
+for each 8 bits of data. Every four bytes of metadata is followed by three
+bytes of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+      - p
+      - p
+    * - start + 7:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-16:
+
+V4L2_META_FMT_GENERIC_CSI2_16
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_16 contains packed 8-bit generic metadata, 16 bits
+for each 8 bits of data. Every byte of metadata is followed by one byte of
+padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - M\ :sub:`10`
+      - p
+      - M\ :sub:`20`
+      - p
+      - M\ :sub:`30`
+      - p
+    * - start + 8:
+      - M\ :sub:`01`
+      - p
+      - M\ :sub:`11`
+      - p
+      - M\ :sub:`21`
+      - p
+      - M\ :sub:`31`
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-20:
+
+V4L2_META_FMT_GENERIC_CSI2_20
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_20 contains packed 8-bit generic metadata, 20 bits
+for each 8 bits of data. Every byte of metadata is followed by alternating one
+and two bytes of padding. The way the data is stored follows the CSI-2
+specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - M\ :sub:`10`
+      - p
+      - p
+      - M\ :sub:`20`
+      - p
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 10:
+      - M\ :sub:`01`
+      - p
+      - M\ :sub:`11`
+      - p
+      - p
+      - M\ :sub:`21`
+      - p
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-24:
+
+V4L2_META_FMT_GENERIC_CSI2_24
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_24 contains packed 8-bit generic metadata, 24 bits
+for each 8 bits of data. Every byte of metadata is followed by two bytes of
+padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - p
+      - M\ :sub:`10`
+      - p
+      - p
+      - M\ :sub:`20`
+      - p
+      - p
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 12:
+      - M\ :sub:`01`
+      - p
+      - p
+      - M\ :sub:`11`
+      - p
+      - p
+      - M\ :sub:`21`
+      - p
+      - p
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-2-24:
+
+V4L2_META_FMT_GENERIC_CSI2_2_24
+-------------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_2_24 contains packed 8-bit generic metadata, 24 bits
+for each two times 8 bits of data. Every two bytes of metadata are followed by
+one byte of padding. The way the data is stored follows the CSI-2
+specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_2_24.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - p
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+    * - start + 6:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - p
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+
drivers/media/v4l2-core/v4l2-ioctl.c +8 -0
index f4d9d6279094..fbbddc333a30 100644
@@ -1452,6 +1452,14 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
+	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8b Generic Meta, 10b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8b Generic Meta, 12b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8b Generic Meta, 14b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8b Generic Meta, 16b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8b Generic Meta, 20b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8b Generic Meta, 24b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_2_24:	descr = "2x8b Generic Meta, 24b CSI-2"; break;
 
 	default:
 		/* Compressed formats */
include/uapi/linux/videodev2.h +9 -0
index 78260e5d9985..b4284a564025 100644
@@ -838,6 +838,15 @@ struct v4l2_pix_format {
 #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
 #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
 
+#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_2_24	v4l2_fourcc('M', 'C', '2', 'O') /* 2 bytes of 8-bit metadata, 24-bit CSI-2 packed */
+
 /* priv field value to indicates that subsequent fields are valid. */
 #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
 
Signature · 3 lines
Regards,

Sakari Ailus
maintainer
#59 lore raw

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

Quoted 78 lines · Sakari Ailus

Hi Laurent,

Quoted 64 lines · Laurent Pinchart
Quoted 50 lines · Sakari Ailus

Generic 8-bit metadata formats define the in-memory data layout but not the format of the data itself. The reasoning for having such formats is to allow CSI-2 receiver drivers to receive and DMA drivers to write the data to memory without knowing a large number of device specific formats.

These formats may be used only in conjunction of a Media controller pipeline where the internal pad of the source sub-device defines the specific format of the data (using an mbus code).

---
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 331 ++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
 include/uapi/linux/videodev2.h                |   9 +
 4 files changed, 349 insertions(+)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
Documentation/userspace-api/media/v4l/meta-formats.rst +1 -0
index 0bb61fc5bc00..919f595576b9 100644
@@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
     metafmt-vsp1-hgo
     metafmt-vsp1-hgt
     metafmt-vivid
+    metafmt-generic
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +17 -0
new file mode 100644
index 000000000000..a27bfc721edf
@@ -0,0 +1,331 @@
+.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
+
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O')
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+
+
+Generic line-based metadata formats
+
+
+Description
+===========
+
+These generic line-based metadata formats define the memory layout of the data
+without defining the format or meaning of the metadata itself. These formats may
+only be used with a Media controller pipeline where the more specific format is
+defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL>` of the source

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 27

Quoted 10 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +10 -0
+sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`.
+
+.. _v4l2-meta-fmt-generic-8:
+
+V4L2_META_FMT_GENERIC_8
+-----------------------
+
+The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
+
+This format is also used on CSI-2 on both 8 bits per sample as well as on

s/also on/by/

Quoted 1 line

I would also mention "MIPI CCS" instead of "CSI-2".

If CCS were to be mentioned here, then all uses of this format should be included as well.

Quoted 6 lines

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 28

Quoted 1 line
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +1 -0
+16 bits per sample when two bytes of metadata are packed into one sample.

"bits per sample" is very ill-defined for metadata, as there's no sample. I would write "for both the RAW8 packing and the 2 bytes RAW16 packing" or something similar.

Quoted 7 lines
Quoted 1 line

Similar comments for below.

From CSI-2 bus point of view there's no difference between pixel and embedded data when it comes to encoding that data. "Sample" is the next best term beyond "pixel", as the bus can carry samples that may or may not be pixel data. But I'm fine with changing the wording if you think it makes it more understandable.

The CSI-2 specification doesn't seem to define any "sample" concept.

Quoted 35 lines
Quoted 26 lines
Quoted 26 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +26 -0
+
+**Byte Order Of V4L2_META_FMT_GENERIC_8.**
+Each cell is one byte. "M" denotes a byte of metadata.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+    * - start + 4:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+
+.. _v4l2-meta-fmt-generic-csi2-10:
+
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
Quoted 8 lines

Documentation/userspace-api/media/v4l/metafmt-generic.rst

Quoted 4 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +4 -0
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding.

It sounds really weird to write that this format writes 10 bits for each 8 bits of data, when essentially it adds a packing byte every four bytes.

Quoted 2 lines

That's how the hardware has been implemented and probably there is a hardware implementation related reason for this.

I can imagine the specification has been designed to make it possible to push embedded data and pixel data through the same serialization hardware, but I can't tell if that's how hardware has been implemented. Still, from an application point of view, when documenting pixel formats, it sounds confusing. Can't we instead say that there's a 0x55 padding after every four bytes of data ?

Quoted 319 lines
Quoted 3 lines
Quoted 1 line

The way the data is stored follows the CSI-2 specification.

Again, it's CCS, not CSI-2.

The packing comes from CSI-2 but CCS adds padding. I'll add this information.

Quoted 312 lines
Quoted 312 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +273 -0
+
+This format is also used on CSI-2 on 20 bits per sample format that packs two
+bytes of metadata into one sample.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+    * - start + 5:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-12:
+
+V4L2_META_FMT_GENERIC_CSI2_12
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_12 contains packed 8-bit generic metadata, 12 bits
+for each 8 bits of data. Every four bytes of metadata is followed by two bytes
+of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 6:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-14:
+
+V4L2_META_FMT_GENERIC_CSI2_14
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_14 contains packed 8-bit generic metadata, 14 bits
+for each 8 bits of data. Every four bytes of metadata is followed by three
+bytes of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+      - p
+      - p
+    * - start + 7:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-16:
+
+V4L2_META_FMT_GENERIC_CSI2_16
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_16 contains packed 8-bit generic metadata, 16 bits
+for each 8 bits of data. Every byte of metadata is followed by one byte of
+padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - M\ :sub:`10`
+      - p
+      - M\ :sub:`20`
+      - p
+      - M\ :sub:`30`
+      - p
+    * - start + 8:
+      - M\ :sub:`01`
+      - p
+      - M\ :sub:`11`
+      - p
+      - M\ :sub:`21`
+      - p
+      - M\ :sub:`31`
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-20:
+
+V4L2_META_FMT_GENERIC_CSI2_20
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_20 contains packed 8-bit generic metadata, 20 bits
+for each 8 bits of data. Every byte of metadata is followed by alternating one
+and two bytes of padding. The way the data is stored follows the CSI-2
+specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - M\ :sub:`10`
+      - p
+      - p
+      - M\ :sub:`20`
+      - p
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 10:
+      - M\ :sub:`01`
+      - p
+      - M\ :sub:`11`
+      - p
+      - p
+      - M\ :sub:`21`
+      - p
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-24:
+
+V4L2_META_FMT_GENERIC_CSI2_24
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_24 contains packed 8-bit generic metadata, 24 bits
+for each 8 bits of data. Every byte of metadata is followed by two bytes of
+padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - p
+      - M\ :sub:`10`
+      - p
+      - p
+      - M\ :sub:`20`
+      - p
+      - p
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 12:
+      - M\ :sub:`01`
+      - p
+      - p
+      - M\ :sub:`11`
+      - p
+      - p
+      - M\ :sub:`21`
+      - p
+      - p
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-2-24:
+
+V4L2_META_FMT_GENERIC_CSI2_2_24
+-------------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_2_24 contains packed 8-bit generic metadata, 24 bits
+for each two times 8 bits of data. Every two bytes of metadata are followed by
+one byte of padding. The way the data is stored follows the CSI-2
+specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_2_24.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - p
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+    * - start + 6:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - p
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+
drivers/media/v4l2-core/v4l2-ioctl.c +8 -0
index f4d9d6279094..fbbddc333a30 100644
@@ -1452,6 +1452,14 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
+	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8b Generic Meta, 10b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8b Generic Meta, 12b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8b Generic Meta, 14b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8b Generic Meta, 16b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8b Generic Meta, 20b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8b Generic Meta, 24b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_2_24:	descr = "2x8b Generic Meta, 24b CSI-2"; break;
 
 	default:
 		/* Compressed formats */
include/uapi/linux/videodev2.h +9 -0
index 78260e5d9985..b4284a564025 100644
@@ -838,6 +838,15 @@ struct v4l2_pix_format {
 #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
 #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
 
+#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_2_24	v4l2_fourcc('M', 'C', '2', 'O') /* 2 bytes of 8-bit metadata, 24-bit CSI-2 packed */
+
 /* priv field value to indicates that subsequent fields are valid. */
 #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
 
Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#60 lore raw

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

Hi Laurent,

Quoted 127 lines · Laurent Pinchart
Quoted 78 lines · Sakari Ailus

Hi Laurent,

Quoted 64 lines · Laurent Pinchart
Quoted 50 lines · Sakari Ailus

Generic 8-bit metadata formats define the in-memory data layout but not the format of the data itself. The reasoning for having such formats is to allow CSI-2 receiver drivers to receive and DMA drivers to write the data to memory without knowing a large number of device specific formats.

These formats may be used only in conjunction of a Media controller pipeline where the internal pad of the source sub-device defines the specific format of the data (using an mbus code).

---
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 331 ++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
 include/uapi/linux/videodev2.h                |   9 +
 4 files changed, 349 insertions(+)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
Documentation/userspace-api/media/v4l/meta-formats.rst +1 -0
index 0bb61fc5bc00..919f595576b9 100644
@@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
     metafmt-vsp1-hgo
     metafmt-vsp1-hgt
     metafmt-vivid
+    metafmt-generic
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +17 -0
new file mode 100644
index 000000000000..a27bfc721edf
@@ -0,0 +1,331 @@
+.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
+
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O')
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+
+
+Generic line-based metadata formats
+
+
+Description
+===========
+
+These generic line-based metadata formats define the memory layout of the data
+without defining the format or meaning of the metadata itself. These formats may
+only be used with a Media controller pipeline where the more specific format is
+defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL>` of the source

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 27

Quoted 10 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +10 -0
+sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`.
+
+.. _v4l2-meta-fmt-generic-8:
+
+V4L2_META_FMT_GENERIC_8
+-----------------------
+
+The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
+
+This format is also used on CSI-2 on both 8 bits per sample as well as on

s/also on/by/

Quoted 1 line

I would also mention "MIPI CCS" instead of "CSI-2".

If CCS were to be mentioned here, then all uses of this format should be included as well.

Quoted 6 lines

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 28

Quoted 1 line
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +1 -0
+16 bits per sample when two bytes of metadata are packed into one sample.

"bits per sample" is very ill-defined for metadata, as there's no sample. I would write "for both the RAW8 packing and the 2 bytes RAW16 packing" or something similar.

Quoted 7 lines
Quoted 1 line

Similar comments for below.

From CSI-2 bus point of view there's no difference between pixel and embedded data when it comes to encoding that data. "Sample" is the next best term beyond "pixel", as the bus can carry samples that may or may not be pixel data. But I'm fine with changing the wording if you think it makes it more understandable.

The CSI-2 specification doesn't seem to define any "sample" concept.

Quoted 35 lines
Quoted 26 lines
Quoted 26 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +26 -0
+
+**Byte Order Of V4L2_META_FMT_GENERIC_8.**
+Each cell is one byte. "M" denotes a byte of metadata.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+    * - start + 4:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+
+.. _v4l2-meta-fmt-generic-csi2-10:
+
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
Quoted 8 lines

Documentation/userspace-api/media/v4l/metafmt-generic.rst

Quoted 4 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +4 -0
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding.

It sounds really weird to write that this format writes 10 bits for each 8 bits of data, when essentially it adds a packing byte every four bytes.

Quoted 2 lines

That's how the hardware has been implemented and probably there is a hardware implementation related reason for this.

Quoted 6 lines

I can imagine the specification has been designed to make it possible to push embedded data and pixel data through the same serialization hardware, but I can't tell if that's how hardware has been implemented. Still, from an application point of view, when documenting pixel formats, it sounds confusing. Can't we instead say that there's a 0x55 padding after every four bytes of data ?

What if we get an implementation that does not use 0x55 value for padding?

It'd be safer not to define the value of the padding byte, it doesn't matter from parsing point in any case.

Signature · 1 line
Sakari Ailus
maintainer
#61 lore raw

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

Quoted 133 lines · Sakari Ailus
Quoted 125 lines · Laurent Pinchart
Quoted 76 lines · Sakari Ailus
Quoted 64 lines · Laurent Pinchart
Quoted 50 lines · Sakari Ailus

Generic 8-bit metadata formats define the in-memory data layout but not the format of the data itself. The reasoning for having such formats is to allow CSI-2 receiver drivers to receive and DMA drivers to write the data to memory without knowing a large number of device specific formats.

These formats may be used only in conjunction of a Media controller pipeline where the internal pad of the source sub-device defines the specific format of the data (using an mbus code).

---
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 331 ++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
 include/uapi/linux/videodev2.h                |   9 +
 4 files changed, 349 insertions(+)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
Documentation/userspace-api/media/v4l/meta-formats.rst +1 -0
index 0bb61fc5bc00..919f595576b9 100644
@@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
     metafmt-vsp1-hgo
     metafmt-vsp1-hgt
     metafmt-vivid
+    metafmt-generic
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +17 -0
new file mode 100644
index 000000000000..a27bfc721edf
@@ -0,0 +1,331 @@
+.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
+
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O')
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+
+
+Generic line-based metadata formats
+
+
+Description
+===========
+
+These generic line-based metadata formats define the memory layout of the data
+without defining the format or meaning of the metadata itself. These formats may
+only be used with a Media controller pipeline where the more specific format is
+defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL>` of the source

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 27

Quoted 10 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +10 -0
+sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`.
+
+.. _v4l2-meta-fmt-generic-8:
+
+V4L2_META_FMT_GENERIC_8
+-----------------------
+
+The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
+
+This format is also used on CSI-2 on both 8 bits per sample as well as on

s/also on/by/

Quoted 1 line

I would also mention "MIPI CCS" instead of "CSI-2".

If CCS were to be mentioned here, then all uses of this format should be included as well.

Quoted 6 lines

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 28

Quoted 1 line
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +1 -0
+16 bits per sample when two bytes of metadata are packed into one sample.

"bits per sample" is very ill-defined for metadata, as there's no sample. I would write "for both the RAW8 packing and the 2 bytes RAW16 packing" or something similar.

Quoted 7 lines
Quoted 1 line

Similar comments for below.

From CSI-2 bus point of view there's no difference between pixel and embedded data when it comes to encoding that data. "Sample" is the next best term beyond "pixel", as the bus can carry samples that may or may not be pixel data. But I'm fine with changing the wording if you think it makes it more understandable.

The CSI-2 specification doesn't seem to define any "sample" concept.

Quoted 35 lines
Quoted 26 lines
Quoted 26 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +26 -0
+
+**Byte Order Of V4L2_META_FMT_GENERIC_8.**
+Each cell is one byte. "M" denotes a byte of metadata.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+    * - start + 4:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+
+.. _v4l2-meta-fmt-generic-csi2-10:
+
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
Quoted 8 lines

Documentation/userspace-api/media/v4l/metafmt-generic.rst

Quoted 4 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +4 -0
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding.

It sounds really weird to write that this format writes 10 bits for each 8 bits of data, when essentially it adds a packing byte every four bytes.

Quoted 2 lines

That's how the hardware has been implemented and probably there is a hardware implementation related reason for this.

Quoted 6 lines

I can imagine the specification has been designed to make it possible to push embedded data and pixel data through the same serialization hardware, but I can't tell if that's how hardware has been implemented. Still, from an application point of view, when documenting pixel formats, it sounds confusing. Can't we instead say that there's a 0x55 padding after every four bytes of data ?

Quoted 4 lines

What if we get an implementation that does not use 0x55 value for padding?

It'd be safer not to define the value of the padding byte, it doesn't matter from parsing point in any case.

I suppose that would be OK. The parser will likely be sensor-specific anyway, so userspace will know what value to expect for the padding byte, even if not conveyed by the format definition.

Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#62 lore raw

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

In reply to Sakari Ailus
Quoted 79 lines · Sakari Ailus

Hi Laurent,

Quoted 64 lines · Laurent Pinchart
Quoted 50 lines · Sakari Ailus

Generic 8-bit metadata formats define the in-memory data layout but not the format of the data itself. The reasoning for having such formats is to allow CSI-2 receiver drivers to receive and DMA drivers to write the data to memory without knowing a large number of device specific formats.

These formats may be used only in conjunction of a Media controller pipeline where the internal pad of the source sub-device defines the specific format of the data (using an mbus code).

---
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 331 ++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
 include/uapi/linux/videodev2.h                |   9 +
 4 files changed, 349 insertions(+)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
Documentation/userspace-api/media/v4l/meta-formats.rst +1 -0
index 0bb61fc5bc00..919f595576b9 100644
@@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
     metafmt-vsp1-hgo
     metafmt-vsp1-hgt
     metafmt-vivid
+    metafmt-generic
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +17 -0
new file mode 100644
index 000000000000..a27bfc721edf
@@ -0,0 +1,331 @@
+.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
+
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O')
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+
+
+Generic line-based metadata formats
+
+
+Description
+===========
+
+These generic line-based metadata formats define the memory layout of the data
+without defining the format or meaning of the metadata itself. These formats may
+only be used with a Media controller pipeline where the more specific format is
+defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL>` of the source

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 27

Quoted 10 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +10 -0
+sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`.
+
+.. _v4l2-meta-fmt-generic-8:
+
+V4L2_META_FMT_GENERIC_8
+-----------------------
+
+The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
+
+This format is also used on CSI-2 on both 8 bits per sample as well as on

s/also on/by/

Quoted 1 line

I would also mention "MIPI CCS" instead of "CSI-2".

If CCS were to be mentioned here, then all uses of this format should be included as well.

Quoted 7 lines

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 28

Quoted 2 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +1 -0
+16 bits per sample when two bytes of metadata are packed into one sample.

"bits per sample" is very ill-defined for metadata, as there's no sample. I would write "for both the RAW8 packing and the 2 bytes RAW16 packing" or something similar.

Quoted 7 lines
Quoted 1 line

Similar comments for below.

From CSI-2 bus point of view there's no difference between pixel and embedded data when it comes to encoding that data. "Sample" is the next best term beyond "pixel", as the bus can carry samples that may or may not be pixel data. But I'm fine with changing the wording if you think it makes it more understandable.

I'll use "data unit" in v4.

Quoted 367 lines
Quoted 28 lines
Quoted 28 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +26 -0
+
+**Byte Order Of V4L2_META_FMT_GENERIC_8.**
+Each cell is one byte. "M" denotes a byte of metadata.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+    * - start + 4:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+
+.. _v4l2-meta-fmt-generic-csi2-10:
+
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
Quoted 8 lines

Documentation/userspace-api/media/v4l/metafmt-generic.rst

Quoted 4 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +4 -0
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding.

It sounds really weird to write that this format writes 10 bits for each 8 bits of data, when essentially it adds a packing byte every four bytes.

That's how the hardware has been implemented and probably there is a hardware implementation related reason for this.

Quoted 4 lines
Quoted 2 lines

The way the data is stored follows the CSI-2 specification.

Again, it's CCS, not CSI-2.

The packing comes from CSI-2 but CCS adds padding. I'll add this information.

Quoted 314 lines
Quoted 313 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +273 -0
+
+This format is also used on CSI-2 on 20 bits per sample format that packs two
+bytes of metadata into one sample.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+    * - start + 5:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-12:
+
+V4L2_META_FMT_GENERIC_CSI2_12
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_12 contains packed 8-bit generic metadata, 12 bits
+for each 8 bits of data. Every four bytes of metadata is followed by two bytes
+of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 6:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-14:
+
+V4L2_META_FMT_GENERIC_CSI2_14
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_14 contains packed 8-bit generic metadata, 14 bits
+for each 8 bits of data. Every four bytes of metadata is followed by three
+bytes of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+      - p
+      - p
+    * - start + 7:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-16:
+
+V4L2_META_FMT_GENERIC_CSI2_16
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_16 contains packed 8-bit generic metadata, 16 bits
+for each 8 bits of data. Every byte of metadata is followed by one byte of
+padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - M\ :sub:`10`
+      - p
+      - M\ :sub:`20`
+      - p
+      - M\ :sub:`30`
+      - p
+    * - start + 8:
+      - M\ :sub:`01`
+      - p
+      - M\ :sub:`11`
+      - p
+      - M\ :sub:`21`
+      - p
+      - M\ :sub:`31`
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-20:
+
+V4L2_META_FMT_GENERIC_CSI2_20
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_20 contains packed 8-bit generic metadata, 20 bits
+for each 8 bits of data. Every byte of metadata is followed by alternating one
+and two bytes of padding. The way the data is stored follows the CSI-2
+specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - M\ :sub:`10`
+      - p
+      - p
+      - M\ :sub:`20`
+      - p
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 10:
+      - M\ :sub:`01`
+      - p
+      - M\ :sub:`11`
+      - p
+      - p
+      - M\ :sub:`21`
+      - p
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-24:
+
+V4L2_META_FMT_GENERIC_CSI2_24
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_24 contains packed 8-bit generic metadata, 24 bits
+for each 8 bits of data. Every byte of metadata is followed by two bytes of
+padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - p
+      - M\ :sub:`10`
+      - p
+      - p
+      - M\ :sub:`20`
+      - p
+      - p
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 12:
+      - M\ :sub:`01`
+      - p
+      - p
+      - M\ :sub:`11`
+      - p
+      - p
+      - M\ :sub:`21`
+      - p
+      - p
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-2-24:
+
+V4L2_META_FMT_GENERIC_CSI2_2_24
+-------------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_2_24 contains packed 8-bit generic metadata, 24 bits
+for each two times 8 bits of data. Every two bytes of metadata are followed by
+one byte of padding. The way the data is stored follows the CSI-2
+specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_2_24.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - p
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+    * - start + 6:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - p
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+
drivers/media/v4l2-core/v4l2-ioctl.c +8 -0
index f4d9d6279094..fbbddc333a30 100644
@@ -1452,6 +1452,14 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
+	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8b Generic Meta, 10b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8b Generic Meta, 12b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8b Generic Meta, 14b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8b Generic Meta, 16b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8b Generic Meta, 20b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8b Generic Meta, 24b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_2_24:	descr = "2x8b Generic Meta, 24b CSI-2"; break;
 
 	default:
 		/* Compressed formats */
include/uapi/linux/videodev2.h +9 -0
index 78260e5d9985..b4284a564025 100644
@@ -838,6 +838,15 @@ struct v4l2_pix_format {
 #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
 #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
 
+#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_2_24	v4l2_fourcc('M', 'C', '2', 'O') /* 2 bytes of 8-bit metadata, 24-bit CSI-2 packed */
+
 /* priv field value to indicates that subsequent fields are valid. */
 #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
 
Signature · 3 lines
Regards,

Sakari Ailus
Signature · 1 line
Sakari Ailus
maintainer
#63 lore raw

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

Quoted 79 lines · Sakari Ailus
Quoted 78 lines · Sakari Ailus
Quoted 64 lines · Laurent Pinchart
Quoted 50 lines · Sakari Ailus

Generic 8-bit metadata formats define the in-memory data layout but not the format of the data itself. The reasoning for having such formats is to allow CSI-2 receiver drivers to receive and DMA drivers to write the data to memory without knowing a large number of device specific formats.

These formats may be used only in conjunction of a Media controller pipeline where the internal pad of the source sub-device defines the specific format of the data (using an mbus code).

---
 .../userspace-api/media/v4l/meta-formats.rst  |   1 +
 .../media/v4l/metafmt-generic.rst             | 331 ++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   8 +
 include/uapi/linux/videodev2.h                |   9 +
 4 files changed, 349 insertions(+)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
Documentation/userspace-api/media/v4l/meta-formats.rst +1 -0
index 0bb61fc5bc00..919f595576b9 100644
@@ -19,3 +19,4 @@ These formats are used for the :ref:`metadata` interface only.
     metafmt-vsp1-hgo
     metafmt-vsp1-hgt
     metafmt-vivid
+    metafmt-generic
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +17 -0
new file mode 100644
index 000000000000..a27bfc721edf
@@ -0,0 +1,331 @@
+.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
+
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+V4L2_META_FMT_GENERIC_8 ('MET8'), V4L2_META_FMT_GENERIC_CSI2_10 ('MC1A'), V4L2_META_FMT_GENERIC_CSI2_12 ('MC1C'), V4L2_META_FMT_GENERIC_CSI2_14 ('MC1E'), V4L2_META_FMT_GENERIC_CSI2_16 ('MC1G'), V4L2_META_FMT_GENERIC_CSI2_20 ('MC1K'), V4L2_META_FMT_GENERIC_CSI2_24 ('MC1O'), V4L2_META_FMT_GENERIC_CSI2_2_24 ('MC2O')
+**************************************************************************************************************************************************************************************************************************************************************************************************************************
+
+
+Generic line-based metadata formats
+
+
+Description
+===========
+
+These generic line-based metadata formats define the memory layout of the data
+without defining the format or meaning of the metadata itself. These formats may
+only be used with a Media controller pipeline where the more specific format is
+defined in an :ref:`internal source pad <MEDIA-PAD-FL-INTERNAL>` of the source

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 27

Quoted 10 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +10 -0
+sub-device. See also :ref:`source routes <v4l2-subdev-source-routes>`.
+
+.. _v4l2-meta-fmt-generic-8:
+
+V4L2_META_FMT_GENERIC_8
+-----------------------
+
+The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
+
+This format is also used on CSI-2 on both 8 bits per sample as well as on

s/also on/by/

Quoted 1 line

I would also mention "MIPI CCS" instead of "CSI-2".

If CCS were to be mentioned here, then all uses of this format should be included as well.

Quoted 7 lines

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 28

Quoted 1 line
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +1 -0
+16 bits per sample when two bytes of metadata are packed into one sample.

"bits per sample" is very ill-defined for metadata, as there's no sample. I would write "for both the RAW8 packing and the 2 bytes RAW16 packing" or something similar.

Similar comments for below.

Quoted 7 lines
Quoted 5 lines

From CSI-2 bus point of view there's no difference between pixel and embedded data when it comes to encoding that data. "Sample" is the next best term beyond "pixel", as the bus can carry samples that may or may not be pixel data. But I'm fine with changing the wording if you think it makes it more understandable.

I'll use "data unit" in v4.

Please define the term :-)

Quoted 359 lines
Quoted 359 lines
Quoted 26 lines
Quoted 26 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +26 -0
+
+**Byte Order Of V4L2_META_FMT_GENERIC_8.**
+Each cell is one byte. "M" denotes a byte of metadata.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+    * - start + 4:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+
+.. _v4l2-meta-fmt-generic-csi2-10:
+
+V4L2_META_FMT_GENERIC_CSI2_10
+-----------------------------
Quoted 8 lines

Documentation/userspace-api/media/v4l/metafmt-generic.rst

Quoted 4 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +4 -0
+
+V4L2_META_FMT_GENERIC_CSI2_10 contains packed 8-bit generic metadata, 10 bits
+for each 8 bits of data. Every four bytes of metadata is followed by a single
+byte of padding.

It sounds really weird to write that this format writes 10 bits for each 8 bits of data, when essentially it adds a packing byte every four bytes.

That's how the hardware has been implemented and probably there is a hardware implementation related reason for this.

Quoted 4 lines
Quoted 2 lines

The way the data is stored follows the CSI-2 specification.

Again, it's CCS, not CSI-2.

The packing comes from CSI-2 but CCS adds padding. I'll add this information.

Quoted 313 lines
Quoted 313 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +273 -0
+
+This format is also used on CSI-2 on 20 bits per sample format that packs two
+bytes of metadata into one sample.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_10.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+    * - start + 5:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-12:
+
+V4L2_META_FMT_GENERIC_CSI2_12
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_12 contains packed 8-bit generic metadata, 12 bits
+for each 8 bits of data. Every four bytes of metadata is followed by two bytes
+of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_12.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 6:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-14:
+
+V4L2_META_FMT_GENERIC_CSI2_14
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_14 contains packed 8-bit generic metadata, 14 bits
+for each 8 bits of data. Every four bytes of metadata is followed by three
+bytes of padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_14.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+      - p
+      - p
+    * - start + 7:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-16:
+
+V4L2_META_FMT_GENERIC_CSI2_16
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_16 contains packed 8-bit generic metadata, 16 bits
+for each 8 bits of data. Every byte of metadata is followed by one byte of
+padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_16.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - M\ :sub:`10`
+      - p
+      - M\ :sub:`20`
+      - p
+      - M\ :sub:`30`
+      - p
+    * - start + 8:
+      - M\ :sub:`01`
+      - p
+      - M\ :sub:`11`
+      - p
+      - M\ :sub:`21`
+      - p
+      - M\ :sub:`31`
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-20:
+
+V4L2_META_FMT_GENERIC_CSI2_20
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_20 contains packed 8-bit generic metadata, 20 bits
+for each 8 bits of data. Every byte of metadata is followed by alternating one
+and two bytes of padding. The way the data is stored follows the CSI-2
+specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_20.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - M\ :sub:`10`
+      - p
+      - p
+      - M\ :sub:`20`
+      - p
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 10:
+      - M\ :sub:`01`
+      - p
+      - M\ :sub:`11`
+      - p
+      - p
+      - M\ :sub:`21`
+      - p
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-24:
+
+V4L2_META_FMT_GENERIC_CSI2_24
+-----------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_24 contains packed 8-bit generic metadata, 24 bits
+for each 8 bits of data. Every byte of metadata is followed by two bytes of
+padding. The way the data is stored follows the CSI-2 specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_24.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|p{1.2cm}|p{.8cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - p
+      - p
+      - M\ :sub:`10`
+      - p
+      - p
+      - M\ :sub:`20`
+      - p
+      - p
+      - M\ :sub:`30`
+      - p
+      - p
+    * - start + 12:
+      - M\ :sub:`01`
+      - p
+      - p
+      - M\ :sub:`11`
+      - p
+      - p
+      - M\ :sub:`21`
+      - p
+      - p
+      - M\ :sub:`31`
+      - p
+      - p
+
+.. _v4l2-meta-fmt-generic-csi2-2-24:
+
+V4L2_META_FMT_GENERIC_CSI2_2_24
+-------------------------------
+
+V4L2_META_FMT_GENERIC_CSI2_2_24 contains packed 8-bit generic metadata, 24 bits
+for each two times 8 bits of data. Every two bytes of metadata are followed by
+one byte of padding. The way the data is stored follows the CSI-2
+specification.
+
+This format is little endian.
+
+**Byte Order Of V4L2_META_FMT_GENERIC_CSI2_2_24.**
+Each cell is one byte. "M" denotes a byte of metadata and "p" a byte of padding.
+
+.. tabularcolumns:: |p{2.4cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|p{1.2cm}|p{1.2cm}|p{.8cm}|
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 12 8 8 8 8 8 8
+
+    * - start + 0:
+      - M\ :sub:`00`
+      - M\ :sub:`10`
+      - p
+      - M\ :sub:`20`
+      - M\ :sub:`30`
+      - p
+    * - start + 6:
+      - M\ :sub:`01`
+      - M\ :sub:`11`
+      - p
+      - M\ :sub:`21`
+      - M\ :sub:`31`
+      - p
+
drivers/media/v4l2-core/v4l2-ioctl.c +8 -0
index f4d9d6279094..fbbddc333a30 100644
@@ -1452,6 +1452,14 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 	case V4L2_PIX_FMT_Y210:		descr = "10-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
+	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8b Generic Meta, 10b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8b Generic Meta, 12b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_14:	descr = "8b Generic Meta, 14b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_16:	descr = "8b Generic Meta, 16b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_20:	descr = "8b Generic Meta, 20b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_24:	descr = "8b Generic Meta, 24b CSI-2"; break;
+	case V4L2_META_FMT_GENERIC_CSI2_2_24:	descr = "2x8b Generic Meta, 24b CSI-2"; break;
 
 	default:
 		/* Compressed formats */
include/uapi/linux/videodev2.h +9 -0
index 78260e5d9985..b4284a564025 100644
@@ -838,6 +838,15 @@ struct v4l2_pix_format {
 #define V4L2_META_FMT_RK_ISP1_PARAMS	v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
 #define V4L2_META_FMT_RK_ISP1_STAT_3A	v4l2_fourcc('R', 'K', '1', 'S') /* Rockchip ISP1 3A Statistics */
 
+#define V4L2_META_FMT_GENERIC_8		v4l2_fourcc('M', 'E', 'T', '8') /* Generic 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_10	v4l2_fourcc('M', 'C', '1', 'A') /* 10-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_12	v4l2_fourcc('M', 'C', '1', 'C') /* 12-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_14	v4l2_fourcc('M', 'C', '1', 'E') /* 14-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_16	v4l2_fourcc('M', 'C', '1', 'G') /* 16-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_20	v4l2_fourcc('M', 'C', '1', 'K') /* 20-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_24	v4l2_fourcc('M', 'C', '1', 'O') /* 24-bit CSI-2 packed 8-bit metadata */
+#define V4L2_META_FMT_GENERIC_CSI2_2_24	v4l2_fourcc('M', 'C', '2', 'O') /* 2 bytes of 8-bit metadata, 24-bit CSI-2 packed */
+
 /* priv field value to indicates that subsequent fields are valid. */
 #define V4L2_PIX_FMT_PRIV_MAGIC		0xfeedcafe
 
Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#64 lore raw

Re: [PATCH v3 07/10] media: uapi: Add generic 8-bit metadata format definitions

Hi Laurent,

Quoted 28 lines · Laurent Pinchart
Quoted 28 lines
Quoted 27 lines
Quoted 10 lines

Documentation/userspace-api/media/v4l/metafmt-generic.rst line 27

Quoted 6 lines
Documentation/userspace-api/media/v4l/metafmt-generic.rst new file +6 -0
+V4L2_META_FMT_GENERIC_8
+-----------------------
+
+The V4L2_META_FMT_GENERIC_8 format is a plain 8-bit metadata format.
+
+This format is also used on CSI-2 on both 8 bits per sample as well as on

s/also on/by/

I would also mention "MIPI CCS" instead of "CSI-2".

If CCS were to be mentioned here, then all uses of this format should be included as well.

Quoted 7 lines
Quoted 1 line

+16 bits per sample when two bytes of metadata are packed into one sample.

"bits per sample" is very ill-defined for metadata, as there's no sample. I would write "for both the RAW8 packing and the 2 bytes RAW16 packing" or something similar.

Similar comments for below.

From CSI-2 bus point of view there's no difference between pixel and embedded data when it comes to encoding that data. "Sample" is the next best term beyond "pixel", as the bus can carry samples that may or may not be pixel data. But I'm fine with changing the wording if you think it makes it more understandable.

Quoted 3 lines
Quoted 1 line

I'll use "data unit" in v4.

Please define the term :-)

To the glossary, how about this:

Data unit

Unit of data transported by a bus. On parallel buses, this is called a sample while on serial buses the data unit is logical. If the data unit is image data, it may also be called a pixel.

Signature · 1 line
Sakari Ailus
author maintainer
#65 lore raw

PATCH v3 8/10 media: v4l: Support line-based metadata capture

In reply to Sakari Ailus

many camera sensors, among other devices, transmit embedded data and image data for each CSI-2 frame. This embedded data typically contains register configuration of the sensor that has been used to capture the image data of the same frame.

The embedded data is received by the CSI-2 receiver and has the same properties as the image data, including that it is line based: it has width, height and bytesperline (stride).

Add these fields to struct v4l2_meta_format and document them.

Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is line-based i.e. these fields of struct v4l2_meta_format are valid for it.

---
 .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
 .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
 .../media/videodev2.h.rst.exceptions              |  1 +
 drivers/media/v4l2-core/v4l2-ioctl.c              |  5 +++--
 include/uapi/linux/videodev2.h                    | 10 ++++++++++
 5 files changed, 36 insertions(+), 2 deletions(-)
Documentation/userspace-api/media/v4l/dev-meta.rst +15 -0
index 0e7e1ee1471a..4b24bae6e171 100644
@@ -65,3 +65,18 @@ to 0.
       - ``buffersize``
       - Maximum buffer size in bytes required for data. The value is set by the
         driver.
+    * - __u32
+      - ``width``
+      - Width of a line of metadata in samples. Valid when :c:type`v4l2_fmtdesc`
+	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``height``
+      - Number of rows of metadata. Valid when :c:type`v4l2_fmtdesc` flag
+	``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``bytesperline``
+      - Offset in bytes between the beginning of two consecutive lines. Valid
+	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
+	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +7 -0
index 000c154b0f98..6d7664345a4e 100644
@@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
 	The application can ask to configure the quantization of the capture
 	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
 	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
+    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
+      - 0x0200
+      - The metadata format is line-based. In this case the ``width``,
+	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
+	valid. The buffer consists of ``height`` lines, each having ``width``
+	bytes of data and offset between the beginning of each two consecutive
+	lines is ``bytesperline``.
 
 Return Value
 ============
Documentation/userspace-api/media/videodev2.h.rst.exceptions +1 -0
index 3e58aac4ef0b..bdc628e8c1d6 100644
@@ -215,6 +215,7 @@ replace define V4L2_FMT_FLAG_CSC_XFER_FUNC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_YCBCR_ENC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_HSV_ENC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_QUANTIZATION fmtdesc-flags
+replace define V4L2_FMT_FLAG_META_LINE_BASED fmtdesc-flags
 
 # V4L2 timecode types
 replace define V4L2_TC_TYPE_24FPS timecode-type
drivers/media/v4l2-core/v4l2-ioctl.c +3 -2
index fbbddc333a30..971d784e7429 100644
@@ -343,8 +343,9 @@ static void v4l_print_format(const void *arg, bool write_only)
 	case V4L2_BUF_TYPE_META_OUTPUT:
 		meta = &p->fmt.meta;
 		pixelformat = meta->dataformat;
-		pr_cont(", dataformat=%p4cc, buffersize=%u\n",
-			&pixelformat, meta->buffersize);
+		pr_cont(", dataformat=%p4cc, buffersize=%u, width=%u, height=%u, bytesperline=%u\n",
+			&pixelformat, meta->buffersize, meta->width,
+			meta->height, meta->bytesperline);
 		break;
 	}
 }
include/uapi/linux/videodev2.h +10 -0
index b4284a564025..d26c0650c6a7 100644
@@ -877,6 +877,7 @@ struct v4l2_fmtdesc {
 #define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0080
 #define V4L2_FMT_FLAG_CSC_HSV_ENC		V4L2_FMT_FLAG_CSC_YCBCR_ENC
 #define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0100
+#define V4L2_FMT_FLAG_META_LINE_BASED		0x0200
 
 	/* Frame Size and frame rate enumeration */
 /*
@@ -2420,10 +2421,19 @@ struct v4l2_sdr_format {
  * struct v4l2_meta_format - metadata format definition
  * @dataformat:		little endian four character code (fourcc)
  * @buffersize:		maximum size in bytes required for data
+ * @width:		number of bytes of data per line (valid for line based
+ *			formats only, see format documentation)
+ * @height:		number of lines of data per buffer (valid for line based
+ *			formats only)
+ * @bytesperline:	offset between the beginnings of two adjacent lines in
+ *			bytes (valid for line based formats only)
  */
 struct v4l2_meta_format {
 	__u32				dataformat;
 	__u32				buffersize;
+	__u32				width;
+	__u32				height;
+	__u32				bytesperline;
 } __attribute__ ((packed));
 
 /**
Signature · 1 line
2.39.2
#66 lore raw

Re: [PATCH v3 08/10] media: v4l: Support line-based metadata capture

Hi Sakari

Quoted 51 lines · Sakari Ailus

many camera sensors, among other devices, transmit embedded data and image data for each CSI-2 frame. This embedded data typically contains register configuration of the sensor that has been used to capture the image data of the same frame.

The embedded data is received by the CSI-2 receiver and has the same properties as the image data, including that it is line based: it has width, height and bytesperline (stride).

Add these fields to struct v4l2_meta_format and document them.

Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is line-based i.e. these fields of struct v4l2_meta_format are valid for it.

---
 .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
 .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
 .../media/videodev2.h.rst.exceptions              |  1 +
 drivers/media/v4l2-core/v4l2-ioctl.c              |  5 +++--
 include/uapi/linux/videodev2.h                    | 10 ++++++++++
 5 files changed, 36 insertions(+), 2 deletions(-)
Documentation/userspace-api/media/v4l/dev-meta.rst +15 -0
index 0e7e1ee1471a..4b24bae6e171 100644
@@ -65,3 +65,18 @@ to 0.
       - ``buffersize``
       - Maximum buffer size in bytes required for data. The value is set by the
         driver.
+    * - __u32
+      - ``width``
+      - Width of a line of metadata in samples. Valid when :c:type`v4l2_fmtdesc`
+	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``height``
+      - Number of rows of metadata. Valid when :c:type`v4l2_fmtdesc` flag
+	``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``bytesperline``
+      - Offset in bytes between the beginning of two consecutive lines. Valid
+	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
+	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +0 -0
index 000c154b0f98..6d7664345a4e 100644

Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst the ``mbus_code`` field is handled differently: · line 235

Quoted 9 lines
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +6 -0
@@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
 	The application can ask to configure the quantization of the capture
 	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
 	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
+    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
+      - 0x0200
+      - The metadata format is line-based. In this case the ``width``,
+	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
+	valid. The buffer consists of ``height`` lines, each having ``width``
+	bytes of data and offset between the beginning of each two consecutive

Isn't ``width`` in samples ?

Quoted 45 lines
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +1 -0
+	lines is ``bytesperline``.
 
 Return Value
 ============
Documentation/userspace-api/media/videodev2.h.rst.exceptions +1 -0
index 3e58aac4ef0b..bdc628e8c1d6 100644
@@ -215,6 +215,7 @@ replace define V4L2_FMT_FLAG_CSC_XFER_FUNC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_YCBCR_ENC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_HSV_ENC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_QUANTIZATION fmtdesc-flags
+replace define V4L2_FMT_FLAG_META_LINE_BASED fmtdesc-flags
 
 # V4L2 timecode types
 replace define V4L2_TC_TYPE_24FPS timecode-type
drivers/media/v4l2-core/v4l2-ioctl.c +3 -2
index fbbddc333a30..971d784e7429 100644
@@ -343,8 +343,9 @@ static void v4l_print_format(const void *arg, bool write_only)
 	case V4L2_BUF_TYPE_META_OUTPUT:
 		meta = &p->fmt.meta;
 		pixelformat = meta->dataformat;
-		pr_cont(", dataformat=%p4cc, buffersize=%u\n",
-			&pixelformat, meta->buffersize);
+		pr_cont(", dataformat=%p4cc, buffersize=%u, width=%u, height=%u, bytesperline=%u\n",
+			&pixelformat, meta->buffersize, meta->width,
+			meta->height, meta->bytesperline);
 		break;
 	}
 }
include/uapi/linux/videodev2.h +1 -0
index b4284a564025..d26c0650c6a7 100644
@@ -877,6 +877,7 @@ struct v4l2_fmtdesc {
 #define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0080
 #define V4L2_FMT_FLAG_CSC_HSV_ENC		V4L2_FMT_FLAG_CSC_YCBCR_ENC
 #define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0100
+#define V4L2_FMT_FLAG_META_LINE_BASED		0x0200
 
 	/* Frame Size and frame rate enumeration */
 /*

include/uapi/linux/videodev2.h struct v4l2_sdr_format { · line 2424

Quoted 4 lines
include/uapi/linux/videodev2.h +1 -0
@@ -2420,10 +2421,19 @@ struct v4l2_sdr_format {
  * struct v4l2_meta_format - metadata format definition
  * @dataformat:		little endian four character code (fourcc)
  * @buffersize:		maximum size in bytes required for data
+ * @width:		number of bytes of data per line (valid for line based

I'm a bit confused here as well, isn't width in samples ?

Quoted 18 lines
include/uapi/linux/videodev2.h +8 -1
+ *			formats only, see format documentation)
+ * @height:		number of lines of data per buffer (valid for line based
+ *			formats only)
+ * @bytesperline:	offset between the beginnings of two adjacent lines in
+ *			bytes (valid for line based formats only)
  */
 struct v4l2_meta_format {
 	__u32				dataformat;
 	__u32				buffersize;
+	__u32				width;
+	__u32				height;
+	__u32				bytesperline;
 } __attribute__ ((packed));
 
 /**
--

2.39.2

author maintainer
#67 lore raw

Re: [PATCH v3 08/10] media: v4l: Support line-based metadata capture

Hi Jacopo,

Quoted 57 lines · Jacopo Mondi

Hi Sakari

Quoted 54 lines · Sakari Ailus

many camera sensors, among other devices, transmit embedded data and image data for each CSI-2 frame. This embedded data typically contains register configuration of the sensor that has been used to capture the image data of the same frame.

The embedded data is received by the CSI-2 receiver and has the same properties as the image data, including that it is line based: it has width, height and bytesperline (stride).

Add these fields to struct v4l2_meta_format and document them.

Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is line-based i.e. these fields of struct v4l2_meta_format are valid for it.

---
 .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
 .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
 .../media/videodev2.h.rst.exceptions              |  1 +
 drivers/media/v4l2-core/v4l2-ioctl.c              |  5 +++--
 include/uapi/linux/videodev2.h                    | 10 ++++++++++
 5 files changed, 36 insertions(+), 2 deletions(-)
Documentation/userspace-api/media/v4l/dev-meta.rst +15 -0
index 0e7e1ee1471a..4b24bae6e171 100644
@@ -65,3 +65,18 @@ to 0.
       - ``buffersize``
       - Maximum buffer size in bytes required for data. The value is set by the
         driver.
+    * - __u32
+      - ``width``
+      - Width of a line of metadata in samples. Valid when :c:type`v4l2_fmtdesc`
+	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``height``
+      - Number of rows of metadata. Valid when :c:type`v4l2_fmtdesc` flag
+	``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``bytesperline``
+      - Offset in bytes between the beginning of two consecutive lines. Valid
+	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
+	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +0 -0
index 000c154b0f98..6d7664345a4e 100644
@@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
 	The application can ask to configure the quantization of the capture
 	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
 	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
Quoted 8 lines

Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst the ``mbus_code`` field is handled differently: · line 235

Quoted 6 lines
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +6 -0
+    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
+      - 0x0200
+      - The metadata format is line-based. In this case the ``width``,
+	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
+	valid. The buffer consists of ``height`` lines, each having ``width``
+	bytes of data and offset between the beginning of each two consecutive

Isn't ``width`` in samples ?

Indeed, it's better to refer to samples for clarity. I'll fix for v4.

I'll also add bytesperline is in bytes (and not in samples).

Quoted 43 lines
Quoted 43 lines
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +1 -0
+	lines is ``bytesperline``.
 
 Return Value
 ============
Documentation/userspace-api/media/videodev2.h.rst.exceptions +1 -0
index 3e58aac4ef0b..bdc628e8c1d6 100644
@@ -215,6 +215,7 @@ replace define V4L2_FMT_FLAG_CSC_XFER_FUNC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_YCBCR_ENC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_HSV_ENC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_QUANTIZATION fmtdesc-flags
+replace define V4L2_FMT_FLAG_META_LINE_BASED fmtdesc-flags
 
 # V4L2 timecode types
 replace define V4L2_TC_TYPE_24FPS timecode-type
drivers/media/v4l2-core/v4l2-ioctl.c +3 -2
index fbbddc333a30..971d784e7429 100644
@@ -343,8 +343,9 @@ static void v4l_print_format(const void *arg, bool write_only)
 	case V4L2_BUF_TYPE_META_OUTPUT:
 		meta = &p->fmt.meta;
 		pixelformat = meta->dataformat;
-		pr_cont(", dataformat=%p4cc, buffersize=%u\n",
-			&pixelformat, meta->buffersize);
+		pr_cont(", dataformat=%p4cc, buffersize=%u, width=%u, height=%u, bytesperline=%u\n",
+			&pixelformat, meta->buffersize, meta->width,
+			meta->height, meta->bytesperline);
 		break;
 	}
 }
include/uapi/linux/videodev2.h +1 -0
index b4284a564025..d26c0650c6a7 100644
@@ -877,6 +877,7 @@ struct v4l2_fmtdesc {
 #define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0080
 #define V4L2_FMT_FLAG_CSC_HSV_ENC		V4L2_FMT_FLAG_CSC_YCBCR_ENC
 #define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0100
+#define V4L2_FMT_FLAG_META_LINE_BASED		0x0200
 
Quoted 9 lines

include/uapi/linux/videodev2.h struct v4l2_sdr_format { · line 2424

Quoted 7 lines
include/uapi/linux/videodev2.h +1 -0
 	/* Frame Size and frame rate enumeration */
 /*
@@ -2420,10 +2421,19 @@ struct v4l2_sdr_format {
  * struct v4l2_meta_format - metadata format definition
  * @dataformat:		little endian four character code (fourcc)
  * @buffersize:		maximum size in bytes required for data
+ * @width:		number of bytes of data per line (valid for line based

I'm a bit confused here as well, isn't width in samples ?

I'll change this one as well.

Quoted 16 lines
Quoted 16 lines
include/uapi/linux/videodev2.h +8 -0
+ *			formats only, see format documentation)
+ * @height:		number of lines of data per buffer (valid for line based
+ *			formats only)
+ * @bytesperline:	offset between the beginnings of two adjacent lines in
+ *			bytes (valid for line based formats only)
  */
 struct v4l2_meta_format {
 	__u32				dataformat;
 	__u32				buffersize;
+	__u32				width;
+	__u32				height;
+	__u32				bytesperline;
 } __attribute__ ((packed));
 
 /**
Signature · 3 lines
Kind regards,

Sakari Ailus
maintainer
#68 lore raw

Re: [PATCH v3 08/10] media: v4l: Support line-based metadata capture

Hi Sakari,

Quoted 3 lines · Sakari Ailus
Quoted 2 lines · Jacopo Mondi
Quoted 1 line · Sakari Ailus

many camera sensors, among other devices, transmit embedded data and image

s/many/Many/

Quoted 62 lines
Quoted 53 lines
Quoted 53 lines

data for each CSI-2 frame. This embedded data typically contains register configuration of the sensor that has been used to capture the image data of the same frame.

The embedded data is received by the CSI-2 receiver and has the same properties as the image data, including that it is line based: it has width, height and bytesperline (stride).

Add these fields to struct v4l2_meta_format and document them.

Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is line-based i.e. these fields of struct v4l2_meta_format are valid for it.

---
 .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
 .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
 .../media/videodev2.h.rst.exceptions              |  1 +
 drivers/media/v4l2-core/v4l2-ioctl.c              |  5 +++--
 include/uapi/linux/videodev2.h                    | 10 ++++++++++
 5 files changed, 36 insertions(+), 2 deletions(-)
Documentation/userspace-api/media/v4l/dev-meta.rst +15 -0
index 0e7e1ee1471a..4b24bae6e171 100644
@@ -65,3 +65,18 @@ to 0.
       - ``buffersize``
       - Maximum buffer size in bytes required for data. The value is set by the
         driver.
+    * - __u32
+      - ``width``
+      - Width of a line of metadata in samples. Valid when :c:type`v4l2_fmtdesc`
+	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``height``
+      - Number of rows of metadata. Valid when :c:type`v4l2_fmtdesc` flag
+	``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``bytesperline``
+      - Offset in bytes between the beginning of two consecutive lines. Valid
+	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
+	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +0 -0
index 000c154b0f98..6d7664345a4e 100644
@@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
 	The application can ask to configure the quantization of the capture
 	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
 	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
Quoted 8 lines

Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst the ``mbus_code`` field is handled differently: · line 235

Quoted 6 lines
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +6 -0
+    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
+      - 0x0200
+      - The metadata format is line-based. In this case the ``width``,
+	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
+	valid. The buffer consists of ``height`` lines, each having ``width``
+	bytes of data and offset between the beginning of each two consecutive

Isn't ``width`` in samples ?

Quoted 1 line

Indeed, it's better to refer to samples for clarity. I'll fix for v4.

How do you define a "sample" in this case ? I wonder if it wouldn't be simpler for both userspace and kernel drivers if the width was specified in bytes, including the padding bytes.

We need an implementation here to test it out. The good news is that I'm working on it, the bad news is that I'm also busy with other things.

Quoted 1 line

I'll also add bytesperline is in bytes (and not in samples).

Thanks for not messing up (more than needed) with my mental health by not specifying bytesperline in something else than bytes :-)

Quoted 69 lines
Quoted 42 lines
Quoted 42 lines
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +1 -0
+	lines is ``bytesperline``.
 
 Return Value
 ============
Documentation/userspace-api/media/videodev2.h.rst.exceptions +1 -0
index 3e58aac4ef0b..bdc628e8c1d6 100644
@@ -215,6 +215,7 @@ replace define V4L2_FMT_FLAG_CSC_XFER_FUNC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_YCBCR_ENC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_HSV_ENC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_QUANTIZATION fmtdesc-flags
+replace define V4L2_FMT_FLAG_META_LINE_BASED fmtdesc-flags
 
 # V4L2 timecode types
 replace define V4L2_TC_TYPE_24FPS timecode-type
drivers/media/v4l2-core/v4l2-ioctl.c +3 -2
index fbbddc333a30..971d784e7429 100644
@@ -343,8 +343,9 @@ static void v4l_print_format(const void *arg, bool write_only)
 	case V4L2_BUF_TYPE_META_OUTPUT:
 		meta = &p->fmt.meta;
 		pixelformat = meta->dataformat;
-		pr_cont(", dataformat=%p4cc, buffersize=%u\n",
-			&pixelformat, meta->buffersize);
+		pr_cont(", dataformat=%p4cc, buffersize=%u, width=%u, height=%u, bytesperline=%u\n",
+			&pixelformat, meta->buffersize, meta->width,
+			meta->height, meta->bytesperline);
 		break;
 	}
 }
include/uapi/linux/videodev2.h +1 -0
index b4284a564025..d26c0650c6a7 100644
@@ -877,6 +877,7 @@ struct v4l2_fmtdesc {
 #define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0080
 #define V4L2_FMT_FLAG_CSC_HSV_ENC		V4L2_FMT_FLAG_CSC_YCBCR_ENC
 #define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0100
+#define V4L2_FMT_FLAG_META_LINE_BASED		0x0200
 
Quoted 9 lines

include/uapi/linux/videodev2.h struct v4l2_sdr_format { · line 2424

Quoted 7 lines
include/uapi/linux/videodev2.h +1 -0
 	/* Frame Size and frame rate enumeration */
 /*
@@ -2420,10 +2421,19 @@ struct v4l2_sdr_format {
  * struct v4l2_meta_format - metadata format definition
  * @dataformat:		little endian four character code (fourcc)
  * @buffersize:		maximum size in bytes required for data
+ * @width:		number of bytes of data per line (valid for line based

I'm a bit confused here as well, isn't width in samples ?

I'll change this one as well.

Quoted 15 lines
Quoted 15 lines
include/uapi/linux/videodev2.h +8 -0
+ *			formats only, see format documentation)
+ * @height:		number of lines of data per buffer (valid for line based
+ *			formats only)
+ * @bytesperline:	offset between the beginnings of two adjacent lines in
+ *			bytes (valid for line based formats only)
  */
 struct v4l2_meta_format {
 	__u32				dataformat;
 	__u32				buffersize;
+	__u32				width;
+	__u32				height;
+	__u32				bytesperline;
 } __attribute__ ((packed));
 
 /**
Signature · 3 lines
Regards,

Laurent Pinchart
#69 lore raw

Re: [PATCH v3 08/10] media: v4l: Support line-based metadata capture

Hi Laurent

Quoted 71 lines · Laurent Pinchart

Hi Sakari,

Quoted 3 lines · Sakari Ailus
Quoted 2 lines · Jacopo Mondi
Quoted 1 line · Sakari Ailus

many camera sensors, among other devices, transmit embedded data and image

s/many/Many/

Quoted 62 lines
Quoted 61 lines
Quoted 50 lines

data for each CSI-2 frame. This embedded data typically contains register configuration of the sensor that has been used to capture the image data of the same frame.

The embedded data is received by the CSI-2 receiver and has the same properties as the image data, including that it is line based: it has width, height and bytesperline (stride).

Add these fields to struct v4l2_meta_format and document them.

Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is line-based i.e. these fields of struct v4l2_meta_format are valid for it.

---
 .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
 .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
 .../media/videodev2.h.rst.exceptions              |  1 +
 drivers/media/v4l2-core/v4l2-ioctl.c              |  5 +++--
 include/uapi/linux/videodev2.h                    | 10 ++++++++++
 5 files changed, 36 insertions(+), 2 deletions(-)
Documentation/userspace-api/media/v4l/dev-meta.rst +15 -0
index 0e7e1ee1471a..4b24bae6e171 100644
@@ -65,3 +65,18 @@ to 0.
       - ``buffersize``
       - Maximum buffer size in bytes required for data. The value is set by the
         driver.
+    * - __u32
+      - ``width``
+      - Width of a line of metadata in samples. Valid when :c:type`v4l2_fmtdesc`
+	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``height``
+      - Number of rows of metadata. Valid when :c:type`v4l2_fmtdesc` flag
+	``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``bytesperline``
+      - Offset in bytes between the beginning of two consecutive lines. Valid
+	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
+	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +0 -0
index 000c154b0f98..6d7664345a4e 100644

Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst the ``mbus_code`` field is handled differently: · line 235

Quoted 9 lines
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +6 -0
@@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
 	The application can ask to configure the quantization of the capture
 	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
 	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
+    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
+      - 0x0200
+      - The metadata format is line-based. In this case the ``width``,
+	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
+	valid. The buffer consists of ``height`` lines, each having ``width``
+	bytes of data and offset between the beginning of each two consecutive

Isn't ``width`` in samples ?

Quoted 5 lines
Quoted 1 line

Indeed, it's better to refer to samples for clarity. I'll fix for v4.

How do you define a "sample" in this case ? I wonder if it wouldn't be simpler for both userspace and kernel drivers if the width was specified in bytes, including the padding bytes.

Wouldn't this make the image line length (expressed in 'samples') different than the embedded data line length ? Would this confuse userspace or is it fine ?

Quoted 9 lines

We need an implementation here to test it out. The good news is that I'm working on it, the bad news is that I'm also busy with other things.

Quoted 1 line

I'll also add bytesperline is in bytes (and not in samples).

Thanks for not messing up (more than needed) with my mental health by not specifying bytesperline in something else than bytes :-)

:)

Quoted 74 lines
Quoted 68 lines
Quoted 42 lines
Quoted 42 lines
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +1 -0
+	lines is ``bytesperline``.
 
 Return Value
 ============
Documentation/userspace-api/media/videodev2.h.rst.exceptions +1 -0
index 3e58aac4ef0b..bdc628e8c1d6 100644
@@ -215,6 +215,7 @@ replace define V4L2_FMT_FLAG_CSC_XFER_FUNC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_YCBCR_ENC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_HSV_ENC fmtdesc-flags
 replace define V4L2_FMT_FLAG_CSC_QUANTIZATION fmtdesc-flags
+replace define V4L2_FMT_FLAG_META_LINE_BASED fmtdesc-flags
 
 # V4L2 timecode types
 replace define V4L2_TC_TYPE_24FPS timecode-type
drivers/media/v4l2-core/v4l2-ioctl.c +3 -2
index fbbddc333a30..971d784e7429 100644
@@ -343,8 +343,9 @@ static void v4l_print_format(const void *arg, bool write_only)
 	case V4L2_BUF_TYPE_META_OUTPUT:
 		meta = &p->fmt.meta;
 		pixelformat = meta->dataformat;
-		pr_cont(", dataformat=%p4cc, buffersize=%u\n",
-			&pixelformat, meta->buffersize);
+		pr_cont(", dataformat=%p4cc, buffersize=%u, width=%u, height=%u, bytesperline=%u\n",
+			&pixelformat, meta->buffersize, meta->width,
+			meta->height, meta->bytesperline);
 		break;
 	}
 }
include/uapi/linux/videodev2.h +1 -0
index b4284a564025..d26c0650c6a7 100644
@@ -877,6 +877,7 @@ struct v4l2_fmtdesc {
 #define V4L2_FMT_FLAG_CSC_YCBCR_ENC		0x0080
 #define V4L2_FMT_FLAG_CSC_HSV_ENC		V4L2_FMT_FLAG_CSC_YCBCR_ENC
 #define V4L2_FMT_FLAG_CSC_QUANTIZATION		0x0100
+#define V4L2_FMT_FLAG_META_LINE_BASED		0x0200
 
Quoted 9 lines

include/uapi/linux/videodev2.h struct v4l2_sdr_format { · line 2424

Quoted 7 lines
include/uapi/linux/videodev2.h +1 -0
 	/* Frame Size and frame rate enumeration */
 /*
@@ -2420,10 +2421,19 @@ struct v4l2_sdr_format {
  * struct v4l2_meta_format - metadata format definition
  * @dataformat:		little endian four character code (fourcc)
  * @buffersize:		maximum size in bytes required for data
+ * @width:		number of bytes of data per line (valid for line based

I'm a bit confused here as well, isn't width in samples ?

I'll change this one as well.

Quoted 14 lines
Quoted 14 lines
include/uapi/linux/videodev2.h +8 -0
+ *			formats only, see format documentation)
+ * @height:		number of lines of data per buffer (valid for line based
+ *			formats only)
+ * @bytesperline:	offset between the beginnings of two adjacent lines in
+ *			bytes (valid for line based formats only)
  */
 struct v4l2_meta_format {
 	__u32				dataformat;
 	__u32				buffersize;
+	__u32				width;
+	__u32				height;
+	__u32				bytesperline;
 } __attribute__ ((packed));
 
Quoted 1 line
Quoted 1 line

include/uapi/linux/videodev2.h struct v4l2_sdr_format { · line 2439

Quoted 1 line
include/uapi/linux/videodev2.h +0 -0
 /**

--
Regards,

Laurent Pinchart

author maintainer
#70 lore raw

Re: [PATCH v3 08/10] media: v4l: Support line-based metadata capture

Hi Laurent,

Quoted 76 lines · Jacopo Mondi

Hi Laurent

Quoted 73 lines · Laurent Pinchart

Hi Sakari,

Quoted 3 lines · Sakari Ailus
Quoted 2 lines · Jacopo Mondi
Quoted 1 line · Sakari Ailus

many camera sensors, among other devices, transmit embedded data and image

s/many/Many/

Quoted 63 lines
Quoted 53 lines
Quoted 53 lines

data for each CSI-2 frame. This embedded data typically contains register configuration of the sensor that has been used to capture the image data of the same frame.

The embedded data is received by the CSI-2 receiver and has the same properties as the image data, including that it is line based: it has width, height and bytesperline (stride).

Add these fields to struct v4l2_meta_format and document them.

Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is line-based i.e. these fields of struct v4l2_meta_format are valid for it.

---
 .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
 .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
 .../media/videodev2.h.rst.exceptions              |  1 +
 drivers/media/v4l2-core/v4l2-ioctl.c              |  5 +++--
 include/uapi/linux/videodev2.h                    | 10 ++++++++++
 5 files changed, 36 insertions(+), 2 deletions(-)
Documentation/userspace-api/media/v4l/dev-meta.rst +15 -0
index 0e7e1ee1471a..4b24bae6e171 100644
@@ -65,3 +65,18 @@ to 0.
       - ``buffersize``
       - Maximum buffer size in bytes required for data. The value is set by the
         driver.
+    * - __u32
+      - ``width``
+      - Width of a line of metadata in samples. Valid when :c:type`v4l2_fmtdesc`
+	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``height``
+      - Number of rows of metadata. Valid when :c:type`v4l2_fmtdesc` flag
+	``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``bytesperline``
+      - Offset in bytes between the beginning of two consecutive lines. Valid
+	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
+	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +0 -0
index 000c154b0f98..6d7664345a4e 100644
@@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
 	The application can ask to configure the quantization of the capture
 	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
 	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
Quoted 8 lines

Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst the ``mbus_code`` field is handled differently: · line 235

Quoted 6 lines
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +6 -0
+    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
+      - 0x0200
+      - The metadata format is line-based. In this case the ``width``,
+	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
+	valid. The buffer consists of ``height`` lines, each having ``width``
+	bytes of data and offset between the beginning of each two consecutive

Isn't ``width`` in samples ?

Indeed, it's better to refer to samples for clarity. I'll fix for v4.

Quoted 7 lines
Quoted 3 lines

How do you define a "sample" in this case ? I wonder if it wouldn't be simpler for both userspace and kernel drivers if the width was specified in bytes, including the padding bytes.

Wouldn't this make the image line length (expressed in 'samples') different than the embedded data line length ? Would this confuse userspace or is it fine ?

If padding is included to width, then the user needs to calculate how many bytes of metadata there is, apart from the padding (which is redundant). That value is provided to the user for this purpose --- just like for image data.

Quoted 5 lines
Quoted 5 lines

We need an implementation here to test it out. The good news is that I'm working on it, the bad news is that I'm also busy with other things.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

I'll also add bytesperline is in bytes (and not in samples).

Thanks for not messing up (more than needed) with my mental health by not specifying bytesperline in something else than bytes :-)

:)

I remember back in the Amiga days when memory was sometimes scarce, nibbles were used. That works when bytes have an even number of bits.

Signature · 3 lines
Regards,

Sakari Ailus
maintainer
#71 lore raw

Re: [PATCH v3 08/10] media: v4l: Support line-based metadata capture

Quoted 77 lines · Sakari Ailus
Quoted 76 lines · Jacopo Mondi
Quoted 74 lines · Laurent Pinchart
Quoted 3 lines · Sakari Ailus
Quoted 2 lines · Jacopo Mondi
Quoted 1 line · Sakari Ailus

many camera sensors, among other devices, transmit embedded data and image

s/many/Many/

Quoted 62 lines
Quoted 53 lines
Quoted 53 lines

data for each CSI-2 frame. This embedded data typically contains register configuration of the sensor that has been used to capture the image data of the same frame.

The embedded data is received by the CSI-2 receiver and has the same properties as the image data, including that it is line based: it has width, height and bytesperline (stride).

Add these fields to struct v4l2_meta_format and document them.

Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is line-based i.e. these fields of struct v4l2_meta_format are valid for it.

---
 .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
 .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
 .../media/videodev2.h.rst.exceptions              |  1 +
 drivers/media/v4l2-core/v4l2-ioctl.c              |  5 +++--
 include/uapi/linux/videodev2.h                    | 10 ++++++++++
 5 files changed, 36 insertions(+), 2 deletions(-)
Documentation/userspace-api/media/v4l/dev-meta.rst +15 -0
index 0e7e1ee1471a..4b24bae6e171 100644
@@ -65,3 +65,18 @@ to 0.
       - ``buffersize``
       - Maximum buffer size in bytes required for data. The value is set by the
         driver.
+    * - __u32
+      - ``width``
+      - Width of a line of metadata in samples. Valid when :c:type`v4l2_fmtdesc`
+	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``height``
+      - Number of rows of metadata. Valid when :c:type`v4l2_fmtdesc` flag
+	``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``bytesperline``
+      - Offset in bytes between the beginning of two consecutive lines. Valid
+	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
+	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +0 -0
index 000c154b0f98..6d7664345a4e 100644
@@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
 	The application can ask to configure the quantization of the capture
 	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
 	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
Quoted 8 lines

Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst the ``mbus_code`` field is handled differently: · line 235

Quoted 6 lines
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +6 -0
+    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
+      - 0x0200
+      - The metadata format is line-based. In this case the ``width``,
+	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
+	valid. The buffer consists of ``height`` lines, each having ``width``
+	bytes of data and offset between the beginning of each two consecutive

Isn't ``width`` in samples ?

Quoted 1 line

Indeed, it's better to refer to samples for clarity. I'll fix for v4.

How do you define a "sample" in this case ? I wonder if it wouldn't be simpler for both userspace and kernel drivers if the width was specified in bytes, including the padding bytes.

Quoted 8 lines
Quoted 3 lines

Wouldn't this make the image line length (expressed in 'samples') different than the embedded data line length ? Would this confuse userspace or is it fine ?

If padding is included to width, then the user needs to calculate how many bytes of metadata there is, apart from the padding (which is redundant). That value is provided to the user for this purpose --- just like for image data.

The bytesperline value includes padding at the end of the line to achieve a particular stride, so that doesn't tell how many bytes to parse. If the width is specified in "samples", then the parser will need to compute the number of bytes it spans, and then parse those bytes.

Quoted 12 lines
Quoted 9 lines
Quoted 7 lines

We need an implementation here to test it out. The good news is that I'm working on it, the bad news is that I'm also busy with other things.

Quoted 1 line

I'll also add bytesperline is in bytes (and not in samples).

Thanks for not messing up (more than needed) with my mental health by not specifying bytesperline in something else than bytes :-)

:)

I remember back in the Amiga days when memory was sometimes scarce, nibbles were used. That works when bytes have an even number of bits.

Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#72 lore raw

Re: [PATCH v3 08/10] media: v4l: Support line-based metadata capture

Hi Laurent,

Quoted 87 lines · Laurent Pinchart
Quoted 77 lines · Sakari Ailus
Quoted 76 lines · Jacopo Mondi
Quoted 74 lines · Laurent Pinchart
Quoted 3 lines · Sakari Ailus
Quoted 2 lines · Jacopo Mondi
Quoted 1 line · Sakari Ailus

many camera sensors, among other devices, transmit embedded data and image

s/many/Many/

Quoted 62 lines
Quoted 53 lines
Quoted 53 lines

data for each CSI-2 frame. This embedded data typically contains register configuration of the sensor that has been used to capture the image data of the same frame.

The embedded data is received by the CSI-2 receiver and has the same properties as the image data, including that it is line based: it has width, height and bytesperline (stride).

Add these fields to struct v4l2_meta_format and document them.

Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is line-based i.e. these fields of struct v4l2_meta_format are valid for it.

---
 .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
 .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
 .../media/videodev2.h.rst.exceptions              |  1 +
 drivers/media/v4l2-core/v4l2-ioctl.c              |  5 +++--
 include/uapi/linux/videodev2.h                    | 10 ++++++++++
 5 files changed, 36 insertions(+), 2 deletions(-)
Documentation/userspace-api/media/v4l/dev-meta.rst +15 -0
index 0e7e1ee1471a..4b24bae6e171 100644
@@ -65,3 +65,18 @@ to 0.
       - ``buffersize``
       - Maximum buffer size in bytes required for data. The value is set by the
         driver.
+    * - __u32
+      - ``width``
+      - Width of a line of metadata in samples. Valid when :c:type`v4l2_fmtdesc`
+	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``height``
+      - Number of rows of metadata. Valid when :c:type`v4l2_fmtdesc` flag
+	``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``bytesperline``
+      - Offset in bytes between the beginning of two consecutive lines. Valid
+	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
+	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +0 -0
index 000c154b0f98..6d7664345a4e 100644
@@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
 	The application can ask to configure the quantization of the capture
 	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
 	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
Quoted 8 lines

Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst the ``mbus_code`` field is handled differently: · line 235

Quoted 6 lines
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +6 -0
+    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
+      - 0x0200
+      - The metadata format is line-based. In this case the ``width``,
+	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
+	valid. The buffer consists of ``height`` lines, each having ``width``
+	bytes of data and offset between the beginning of each two consecutive

Isn't ``width`` in samples ?

Quoted 1 line

Indeed, it's better to refer to samples for clarity. I'll fix for v4.

How do you define a "sample" in this case ? I wonder if it wouldn't be simpler for both userspace and kernel drivers if the width was specified in bytes, including the padding bytes.

Quoted 8 lines
Quoted 3 lines

Wouldn't this make the image line length (expressed in 'samples') different than the embedded data line length ? Would this confuse userspace or is it fine ?

If padding is included to width, then the user needs to calculate how many bytes of metadata there is, apart from the padding (which is redundant). That value is provided to the user for this purpose --- just like for image data.

Quoted 4 lines

The bytesperline value includes padding at the end of the line to achieve a particular stride, so that doesn't tell how many bytes to parse. If the width is specified in "samples", then the parser will need to compute the number of bytes it spans, and then parse those bytes.

The parser is interested in metadata bytes only, not the padding. The padding should be skipped by the data access function in order to avoid complicating the parser with different padding options.

That's why width should be in data units (bytes of metadata without padding).

Signature · 3 lines
Regards,

Sakari Ailus
author maintainer
#73 lore raw

Re: [PATCH v3 08/10] media: v4l: Support line-based metadata capture

In reply to Sakari Ailus
Quoted 83 lines · Sakari Ailus

Hi Laurent,

Quoted 80 lines · Jacopo Mondi

Hi Laurent

Quoted 76 lines · Laurent Pinchart

Hi Sakari,

Quoted 3 lines · Sakari Ailus
Quoted 2 lines · Jacopo Mondi
Quoted 1 line · Sakari Ailus

many camera sensors, among other devices, transmit embedded data and image

s/many/Many/

Quoted 62 lines
Quoted 53 lines
Quoted 53 lines

data for each CSI-2 frame. This embedded data typically contains register configuration of the sensor that has been used to capture the image data of the same frame.

The embedded data is received by the CSI-2 receiver and has the same properties as the image data, including that it is line based: it has width, height and bytesperline (stride).

Add these fields to struct v4l2_meta_format and document them.

Also add V4L2_FMT_FLAG_META_LINE_BASED to tell a given format is line-based i.e. these fields of struct v4l2_meta_format are valid for it.

---
 .../userspace-api/media/v4l/dev-meta.rst          | 15 +++++++++++++++
 .../userspace-api/media/v4l/vidioc-enum-fmt.rst   |  7 +++++++
 .../media/videodev2.h.rst.exceptions              |  1 +
 drivers/media/v4l2-core/v4l2-ioctl.c              |  5 +++--
 include/uapi/linux/videodev2.h                    | 10 ++++++++++
 5 files changed, 36 insertions(+), 2 deletions(-)
Documentation/userspace-api/media/v4l/dev-meta.rst +15 -0
index 0e7e1ee1471a..4b24bae6e171 100644
@@ -65,3 +65,18 @@ to 0.
       - ``buffersize``
       - Maximum buffer size in bytes required for data. The value is set by the
         driver.
+    * - __u32
+      - ``width``
+      - Width of a line of metadata in samples. Valid when :c:type`v4l2_fmtdesc`
+	flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``height``
+      - Number of rows of metadata. Valid when :c:type`v4l2_fmtdesc` flag
+	``V4L2_FMT_FLAG_META_LINE_BASED`` is set, otherwise zero. See
+	:c:func:`VIDIOC_ENUM_FMT`.
+    * - __u32
+      - ``bytesperline``
+      - Offset in bytes between the beginning of two consecutive lines. Valid
+	when :c:type`v4l2_fmtdesc` flag ``V4L2_FMT_FLAG_META_LINE_BASED`` is
+	set, otherwise zero. See :c:func:`VIDIOC_ENUM_FMT`.
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +0 -0
index 000c154b0f98..6d7664345a4e 100644
@@ -227,6 +227,13 @@ the ``mbus_code`` field is handled differently:
 	The application can ask to configure the quantization of the capture
 	device when calling the :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl with
 	:ref:`V4L2_PIX_FMT_FLAG_SET_CSC <v4l2-pix-fmt-flag-set-csc>` set.
Quoted 8 lines

Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst the ``mbus_code`` field is handled differently: · line 235

Quoted 6 lines
Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +6 -0
+    * - ``V4L2_FMT_FLAG_META_LINE_BASED``
+      - 0x0200
+      - The metadata format is line-based. In this case the ``width``,
+	``height`` and ``bytesperline`` fields of :c:type:`v4l2_meta_format` are
+	valid. The buffer consists of ``height`` lines, each having ``width``
+	bytes of data and offset between the beginning of each two consecutive

Isn't ``width`` in samples ?

Quoted 1 line

Indeed, it's better to refer to samples for clarity. I'll fix for v4.

How do you define a "sample" in this case ? I wonder if it wouldn't be simpler for both userspace and kernel drivers if the width was specified in bytes, including the padding bytes.

Quoted 8 lines
Quoted 3 lines

Wouldn't this make the image line length (expressed in 'samples') different than the embedded data line length ? Would this confuse userspace or is it fine ?

If padding is included to width, then the user needs to calculate how many bytes of metadata there is, apart from the padding (which is redundant). That value is provided to the user for this purpose --- just like for image data.

Think of this: the embedded data parser is split into two, the access function that accesses metadata bytes depending on the generic V4L2 metadata format and the rest, which cares about the specific mbus code. The latter will need the information how many bytes of metadata there is, instead it calculates the position in the buffer based on bytesperline, width and the format itself.

Of course, the amount of padding can be calculated from the total amount, by adding a second function that does it, but still it's somehow backwards way to do that. (Makes me think of negabinaries.)

Signature · 3 lines
Regards,

Sakari Ailus
author maintainer
#74 lore raw

PATCH v3 9/10 media: Add media bus codes for MIPI CCS embedded data

In reply to Sakari Ailus

Add new MIPI CCS embedded data media bus formats.

---
 .../media/v4l/subdev-formats.rst              | 32 +++++++++++++++++++
 include/uapi/linux/media-bus-format.h         | 10 +++++-
 2 files changed, 41 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/v4l/subdev-formats.rst +32 -0
index c615da08502d..5d5407738af9 100644
@@ -8491,3 +8491,35 @@ and finally the bit number in subscript. "p" indicates a padding bit.
       - p
       - p
       - p
+
+MIPI CCS Embedded Data Formats
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+`MIPI CCS <https://www.mipi.org/specifications/camera-command-set>`_ defines an
+metadata format for sensor embedded data, which is used to store the register
+configuration used for capturing a given frame. The format is defined in the CCS
+specification.
+
+The bit depth of the CCS embedded data matches the pixel data bit depth
+configured on the sensor. The formats used and their corresponding generic
+formats are listed in the table below.
+
+.. flat-table: CCS embedded data mbus formats and corresponding generic formats
+    :header-rows: 1
+
+    * - CCS embedded data mbus format
+      - Generic metadata format
+    * - MEDIA_BUS_FMT_CCS_EMBEDDED_8
+      - MEDIA_BUS_FMT_META_8
+    * - MEDIA_BUS_FMT_CCS_EMBEDDED_10
+      - MEDIA_BUS_FMT_META_10
+    * - MEDIA_BUS_FMT_CCS_EMBEDDED_12
+      - MEDIA_BUS_FMT_META_12
+    * - MEDIA_BUS_FMT_CCS_EMBEDDED_14
+      - MEDIA_BUS_FMT_META_14
+    * - MEDIA_BUS_FMT_CCS_EMBEDDED_16
+      - MEDIA_BUS_FMT_META_16
+    * - MEDIA_BUS_FMT_CCS_EMBEDDED_20
+      - MEDIA_BUS_FMT_META_20
+    * - MEDIA_BUS_FMT_CCS_EMBEDDED_24
+      - MEDIA_BUS_FMT_META_24
include/uapi/linux/media-bus-format.h +9 -1
index 2486b4178c5f..31d532446b23 100644
@@ -182,7 +182,15 @@
 #define MEDIA_BUS_FMT_META_20			0x8006
 #define MEDIA_BUS_FMT_META_24			0x8007
 
+/* Specific metadata formats. Next is 0x9008. */
+#define MEDIA_BUS_FMT_CCS_EMBEDDED_8		0x9001
+#define MEDIA_BUS_FMT_CCS_EMBEDDED_10		0x9002
+#define MEDIA_BUS_FMT_CCS_EMBEDDED_12		0x9003
+#define MEDIA_BUS_FMT_CCS_EMBEDDED_14		0x9004
+#define MEDIA_BUS_FMT_CCS_EMBEDDED_16		0x9005
+#define MEDIA_BUS_FMT_CCS_EMBEDDED_20		0x9006
+#define MEDIA_BUS_FMT_CCS_EMBEDDED_24		0x9007
+
 #define MEDIA_BUS_FMT_IS_META(code)		\
 	((code) & 0xf000 == 0x7000 || (code) & 0xf000 == 0x8000)
-
 #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
Signature · 1 line
2.39.2
maintainer
#75 lore raw

Re: [PATCH v3 09/10] media: Add media bus codes for MIPI CCS embedded data

Hi Sakari,

Thank you for the patch.

Quoted 13 lines · Sakari Ailus

Add new MIPI CCS embedded data media bus formats.

---
 .../media/v4l/subdev-formats.rst              | 32 +++++++++++++++++++
 include/uapi/linux/media-bus-format.h         | 10 +++++-
 2 files changed, 41 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/v4l/subdev-formats.rst +0 -0
index c615da08502d..5d5407738af9 100644

Documentation/userspace-api/media/v4l/subdev-formats.rst and finally the bit number in subscript. "p" indicates a padding bit. · line 8498

Quoted 8 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +5 -0
@@ -8491,3 +8491,35 @@ and finally the bit number in subscript. "p" indicates a padding bit.
       - p
       - p
       - p
+
+MIPI CCS Embedded Data Formats
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+`MIPI CCS <https://www.mipi.org/specifications/camera-command-set>`_ defines an

s/an$/a/

Documentation/userspace-api/media/v4l/subdev-formats.rst and finally the bit number in subscript. "p" indicates a padding bit. · line 8501

Quoted 3 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +3 -0
+metadata format for sensor embedded data, which is used to store the register
+configuration used for capturing a given frame. The format is defined in the CCS
+specification.

Strictly speaking, the MIPI CCS embedded data format specifies not just the data packing (insertion of padding bytes) and the data encoding (the data format byte and the tag codes), but also the register addresses and values that are reported in the embedded data. Do you envision the media bus formats defined here as being applicable to sensors that use the same packing and encoding as CCS, but different registers, or only to fully compliant CCS sensors ?

Quoted 45 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +24 -0
+
+The bit depth of the CCS embedded data matches the pixel data bit depth
+configured on the sensor. The formats used and their corresponding generic
+formats are listed in the table below.
+
+.. flat-table: CCS embedded data mbus formats and corresponding generic formats
+    :header-rows: 1
+
+    * - CCS embedded data mbus format
+      - Generic metadata format
+    * - MEDIA_BUS_FMT_CCS_EMBEDDED_8
+      - MEDIA_BUS_FMT_META_8
+    * - MEDIA_BUS_FMT_CCS_EMBEDDED_10
+      - MEDIA_BUS_FMT_META_10
+    * - MEDIA_BUS_FMT_CCS_EMBEDDED_12
+      - MEDIA_BUS_FMT_META_12
+    * - MEDIA_BUS_FMT_CCS_EMBEDDED_14
+      - MEDIA_BUS_FMT_META_14
+    * - MEDIA_BUS_FMT_CCS_EMBEDDED_16
+      - MEDIA_BUS_FMT_META_16
+    * - MEDIA_BUS_FMT_CCS_EMBEDDED_20
+      - MEDIA_BUS_FMT_META_20
+    * - MEDIA_BUS_FMT_CCS_EMBEDDED_24
+      - MEDIA_BUS_FMT_META_24
include/uapi/linux/media-bus-format.h +9 -1
index 2486b4178c5f..31d532446b23 100644
@@ -182,7 +182,15 @@
 #define MEDIA_BUS_FMT_META_20			0x8006
 #define MEDIA_BUS_FMT_META_24			0x8007
 
+/* Specific metadata formats. Next is 0x9008. */
+#define MEDIA_BUS_FMT_CCS_EMBEDDED_8		0x9001
+#define MEDIA_BUS_FMT_CCS_EMBEDDED_10		0x9002
+#define MEDIA_BUS_FMT_CCS_EMBEDDED_12		0x9003
+#define MEDIA_BUS_FMT_CCS_EMBEDDED_14		0x9004
+#define MEDIA_BUS_FMT_CCS_EMBEDDED_16		0x9005
+#define MEDIA_BUS_FMT_CCS_EMBEDDED_20		0x9006
+#define MEDIA_BUS_FMT_CCS_EMBEDDED_24		0x9007
+
 #define MEDIA_BUS_FMT_IS_META(code)		\
 	((code) & 0xf000 == 0x7000 || (code) & 0xf000 == 0x8000)
-
 #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#76 lore raw

Re: [PATCH v3 09/10] media: Add media bus codes for MIPI CCS embedded data

Hi Laurent,

Quoted 13 lines · Laurent Pinchart

Hi Sakari,

Thank you for the patch.

Quoted 8 lines · Sakari Ailus

Add new MIPI CCS embedded data media bus formats.

---
 .../media/v4l/subdev-formats.rst              | 32 +++++++++++++++++++
 include/uapi/linux/media-bus-format.h         | 10 +++++-
 2 files changed, 41 insertions(+), 1 deletion(-)
Quoted 15 lines

Documentation/userspace-api/media/v4l/subdev-formats.rst and finally the bit number in subscript. "p" indicates a padding bit. · line 8498

Quoted 13 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +5 -0
index c615da08502d..5d5407738af9 100644
@@ -8491,3 +8491,35 @@ and finally the bit number in subscript. "p" indicates a padding bit.
       - p
       - p
       - p
+
+MIPI CCS Embedded Data Formats
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+`MIPI CCS <https://www.mipi.org/specifications/camera-command-set>`_ defines an

s/an$/a/

Yes. I think I had "embedded" there in the past...

Quoted 12 lines

Documentation/userspace-api/media/v4l/subdev-formats.rst and finally the bit number in subscript. "p" indicates a padding bit. · line 8501

Quoted 4 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +3 -0
+metadata format for sensor embedded data, which is used to store the register
+configuration used for capturing a given frame. The format is defined in the CCS
+specification.

Strictly speaking, the MIPI CCS embedded data format specifies not just the data packing (insertion of padding bytes) and the data encoding (the data format byte and the tag codes), but also the register addresses and values that are reported in the embedded data. Do you envision the media bus formats defined here as being applicable to sensors that use the same packing and encoding as CCS, but different registers, or only to fully compliant CCS sensors ?

There are sensors that aren't fully compatible with CCS (including those compatible with SMIA and SMIA++) but I wouldn't expect the format to be used by devices that are entirely incompatible with CCS.

Signature · 3 lines
Regards,

Sakari Ailus
maintainer
#77 lore raw

Re: [PATCH v3 09/10] media: Add media bus codes for MIPI CCS embedded data

Quoted 40 lines · Sakari Ailus
Quoted 9 lines · Laurent Pinchart
Quoted 8 lines · Sakari Ailus

Add new MIPI CCS embedded data media bus formats.

---
 .../media/v4l/subdev-formats.rst              | 32 +++++++++++++++++++
 include/uapi/linux/media-bus-format.h         | 10 +++++-
 2 files changed, 41 insertions(+), 1 deletion(-)
Quoted 15 lines

Documentation/userspace-api/media/v4l/subdev-formats.rst and finally the bit number in subscript. "p" indicates a padding bit. · line 8498

Quoted 13 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +5 -0
index c615da08502d..5d5407738af9 100644
@@ -8491,3 +8491,35 @@ and finally the bit number in subscript. "p" indicates a padding bit.
       - p
       - p
       - p
+
+MIPI CCS Embedded Data Formats
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+`MIPI CCS <https://www.mipi.org/specifications/camera-command-set>`_ defines an

s/an$/a/

Yes. I think I had "embedded" there in the past...

Quoted 11 lines

Documentation/userspace-api/media/v4l/subdev-formats.rst and finally the bit number in subscript. "p" indicates a padding bit. · line 8501

Quoted 3 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +3 -0
+metadata format for sensor embedded data, which is used to store the register
+configuration used for capturing a given frame. The format is defined in the CCS
+specification.

Strictly speaking, the MIPI CCS embedded data format specifies not just the data packing (insertion of padding bytes) and the data encoding (the data format byte and the tag codes), but also the register addresses and values that are reported in the embedded data. Do you envision the media bus formats defined here as being applicable to sensors that use the same packing and encoding as CCS, but different registers, or only to fully compliant CCS sensors ?

Quoted 3 lines

There are sensors that aren't fully compatible with CCS (including those compatible with SMIA and SMIA++) but I wouldn't expect the format to be used by devices that are entirely incompatible with CCS.

So if a sensor uses the same packing and encoding as CCS, but a different register set, you would require other media bus codes ? If so, how would you require those media bus codes to be documented ? The packing and encoding could be documented as being identical to CCS, but what about the data itself ? Would you require the datasheet to be public ? Or the entire register set being documented in the V4L2 spec ?

Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#78 lore raw

Re: [PATCH v3 09/10] media: Add media bus codes for MIPI CCS embedded data

Hi Laurent,

Quoted 45 lines · Laurent Pinchart
Quoted 40 lines · Sakari Ailus
Quoted 9 lines · Laurent Pinchart
Quoted 8 lines · Sakari Ailus

Add new MIPI CCS embedded data media bus formats.

---
 .../media/v4l/subdev-formats.rst              | 32 +++++++++++++++++++
 include/uapi/linux/media-bus-format.h         | 10 +++++-
 2 files changed, 41 insertions(+), 1 deletion(-)
Quoted 15 lines

Documentation/userspace-api/media/v4l/subdev-formats.rst and finally the bit number in subscript. "p" indicates a padding bit. · line 8498

Quoted 13 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +5 -0
index c615da08502d..5d5407738af9 100644
@@ -8491,3 +8491,35 @@ and finally the bit number in subscript. "p" indicates a padding bit.
       - p
       - p
       - p
+
+MIPI CCS Embedded Data Formats
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+`MIPI CCS <https://www.mipi.org/specifications/camera-command-set>`_ defines an

s/an$/a/

Yes. I think I had "embedded" there in the past...

Quoted 11 lines

Documentation/userspace-api/media/v4l/subdev-formats.rst and finally the bit number in subscript. "p" indicates a padding bit. · line 8501

Quoted 3 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +3 -0
+metadata format for sensor embedded data, which is used to store the register
+configuration used for capturing a given frame. The format is defined in the CCS
+specification.

Strictly speaking, the MIPI CCS embedded data format specifies not just the data packing (insertion of padding bytes) and the data encoding (the data format byte and the tag codes), but also the register addresses and values that are reported in the embedded data. Do you envision the media bus formats defined here as being applicable to sensors that use the same packing and encoding as CCS, but different registers, or only to fully compliant CCS sensors ?

Quoted 3 lines

There are sensors that aren't fully compatible with CCS (including those compatible with SMIA and SMIA++) but I wouldn't expect the format to be used by devices that are entirely incompatible with CCS.

Quoted 6 lines

So if a sensor uses the same packing and encoding as CCS, but a different register set, you would require other media bus codes ? If so, how would you require those media bus codes to be documented ? The packing and encoding could be documented as being identical to CCS, but what about the data itself ? Would you require the datasheet to be public ? Or the entire register set being documented in the V4L2 spec ?

I'd hope the registers the values of which are written in the embedded data should be documented, either in V4L2 spec or other documentation.

But then, what do you do if you don't have documentation for all the registers while you'd like to use what in general is needed there, exposure time and gain? Do we say no, you can't do that because everything isn't documented, or do we say yes, you can document this format partially. That's what we should discuss and decide.

Note that commonly the case is that whoever would like to use the information in the embedded data may well not have the full documentation either. Also, even without documentation, it is in the vast majority of the cases trivial to figure out where these values can be found.

Signature · 3 lines
Regards,

Sakari Ailus
author maintainer
#79 lore raw

PATCH v3 10/10 media: uapi: v4l: Document source routes

In reply to Sakari Ailus

Document how internal pads are used on source routes.

---
 .../userspace-api/media/v4l/dev-subdev.rst    | 179 ++++++++++++++++++
 1 file changed, 179 insertions(+)
Documentation/userspace-api/media/v4l/dev-subdev.rst +179 -0
index b023918177b5..27b0fe2dc83a 100644
@@ -551,6 +551,27 @@ A stream at a specific point in the media pipeline is identified by the
 sub-device and a (pad, stream) pair. For sub-devices that do not support
 multiplexed streams the 'stream' field is always 0.
 
+.. _v4l2-subdev-source-routes:
+
+Internal pads and source routes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Cases where a single sub-device source pad is traversed by multiple streams one
+or more of which originate from within the sub-device itself are special as
+there is no external sink pad for such routes. In those cases, the sources of
+the internally generated streams are represented by internal sink pads, which
+are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`
+pad flag set.
+
+Internal pads have all the properties of an external pad, including formats and
+selections. The format in this case is the source format of the stream. An
+internal pad always has a single stream only (0).
+
+/Source routes/ are routes from an internal sink pad to a(n external) source
+pad. Generally source routes are not modifiable but they can be activated and
+deactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
+<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
+
 Interaction between routes, streams, formats and selections
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -666,3 +687,161 @@ A common way to accomplish this is to start from the sensors and propagate the
 configurations along the stream towards the receiver,
 using :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
 stream endpoint in each sub-device.
+
+Internal pads setup example
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A simple example of a multiplexed stream setup might be as follows:
+
+- A CCS camera sensor source sub-device, with one sink pad (0), one source pad
+  (1), an internal sink pad (2) that represents the source of embedded
+  data. There are two routes, one from the sink pad to the source, and another
+  from the internal sink pad to the source pad. The embedded data stream needs
+  to be enabled by activating the related route. The configuration of the rest
+  of the CCS sub-devices is omitted from this example.
+
+- Multiplexer bridge (Bridge). The bridge has one sink pad, connected to the
+  sensor (pad 0), and one source pad (pad 1), which outputs two streams.
+
+- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
+  connected to the bridge, and two source pads (pads 1-2), going to the DMA
+  engine. The receiver demultiplexes the incoming streams to the source pads.
+
+- DMA Engines in the SoC (DMA Engine), one for each stream. Each DMA engine is
+  connected to a single source pad in the receiver.
+
+The sensor, the bridge and the receiver are modeled as V4L2 sub-devices,
+exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
+modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
+
+To configure this pipeline, the userspace must take the following steps:
+
+1) Set up media links between entities: connect the sensors to the bridge,
+   bridge to the receiver, and the receiver to the DMA engines. This step does
+   not differ from normal non-multiplexed media controller setup.
+
+2) Configure routing
+
+.. flat-table:: Camera sensor
+    :header-rows: 1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from the sink pad
+    * - 2/0
+      - 1/1
+      - **V4L2_SUBDEV_ROUTE_FL_ACTIVE**
+      - Metadata stream from the internal sink pad
+
+.. flat-table:: Bridge routing table
+    :header-rows: 1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from camera sensor
+    * - 0/1
+      - 1/1
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from camera sensor
+
+.. flat-table:: Receiver routing table
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from camera sensor
+    * - 0/1
+      - 2/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from camera sensor
+
+The options available in sensor's routing configuration are dictated by hardware
+capabilities: typically camera sensors always produce image data stream while
+the embedded data stream typically can be either enabled or disabled.
+
+3) Configure formats and selections
+
+This example assumes that the formats are propagated from sink pad to the source
+pad as-is. The tables contain fields of both struct v4l2_subdev_format and
+struct v4l2_mbus_framefmt. The full configuration of CCS camera sensor is out of
+scope of this example.
+
+.. flat-table:: Formats set on the sub-devices. Bold values are set, others are
+                static or propagated.
+    :header-rows: 1
+    :fill-cells:
+
+    * - Sub-device
+      - Pad/Stream
+      - Width
+      - Height
+      - Code
+    * - :rspan:`3` Camera sensor sub-device (CCS source sub-device)
+      - 0/0
+      - 640
+      - 480
+      - MEDIA_BUS_FMT_SGRBG10
+    * - 1/0
+      - 640
+      - 480
+      - **MEDIA_BUS_FMT_SGRBG10**
+    * - 2/0
+      - 640
+      - 2
+      - MEDIA_BUS_FMT_CCS_EMBEDDED_10
+    * - 1/1
+      - 640
+      - 2
+      - MEDIA_BUS_FMT_META_10
+    * - :rspan:`3` Bridge
+      - 0/0
+      - **640**
+      - **480**
+      - **MEDIA_BUS_FMT_SGRBG10**
+    * - 1/0
+      - 640
+      - 480
+      - MEDIA_BUS_FMT_SGRBG10
+    * - 0/1
+      - **640**
+      - **2**
+      - **MEDIA_BUS_FMT_META_10**
+    * - 1/1
+      - 640
+      - 2
+      - MEDIA_BUS_FMT_META_10
+    * - :rspan:`3` Receiver
+      - 0/0
+      - **640**
+      - **480**
+      - **MEDIA_BUS_FMT_SGRBG10**
+    * - 1/0
+      - 640
+      - 480
+      - MEDIA_BUS_FMT_SGRBG10
+    * - 0/1
+      - **640**
+      - **2**
+      - **MEDIA_BUS_FMT_META_10**
+    * - 2/0
+      - 640
+      - 2
+      - MEDIA_BUS_FMT_META_10
+
+The embedded data format does not need to be configured as the format is
+dictated by the pixel data format in this case.
Signature · 1 line
2.39.2
#80 lore raw

Re: [PATCH v3 10/10] media: uapi: v4l: Document source routes

Quoted 22 lines · Sakari Ailus

Document how internal pads are used on source routes.

---
 .../userspace-api/media/v4l/dev-subdev.rst    | 179 ++++++++++++++++++
 1 file changed, 179 insertions(+)
Documentation/userspace-api/media/v4l/dev-subdev.rst +7 -0
index b023918177b5..27b0fe2dc83a 100644
@@ -551,6 +551,27 @@ A stream at a specific point in the media pipeline is identified by the
 sub-device and a (pad, stream) pair. For sub-devices that do not support
 multiplexed streams the 'stream' field is always 0.
 
+.. _v4l2-subdev-source-routes:
+
+Internal pads and source routes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Cases where a single sub-device source pad is traversed by multiple streams one
+or more of which originate from within the sub-device itself are special as

Documentation/userspace-api/media/v4l/dev-subdev.rst A stream at a specific point in the media pipeline is identified by the · line 570

Quoted 10 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +10 -0
+there is no external sink pad for such routes. In those cases, the sources of
+the internally generated streams are represented by internal sink pads, which
+are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`
+pad flag set.
+
+Internal pads have all the properties of an external pad, including formats and
+selections. The format in this case is the source format of the stream. An
+internal pad always has a single stream only (0).
+
+/Source routes/ are routes from an internal sink pad to a(n external) source

'/Source routes/' appears just like that in the generated documentation, that's probably not what you intended.

Quoted 8 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +4 -0
+pad. Generally source routes are not modifiable but they can be activated and
+deactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
+<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
+
 Interaction between routes, streams, formats and selections
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

Documentation/userspace-api/media/v4l/dev-subdev.rst A common way to accomplish this is to start from the sensors and propagate the · line 696

Quoted 10 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +7 -0
@@ -666,3 +687,161 @@ A common way to accomplish this is to start from the sensors and propagate the
 configurations along the stream towards the receiver,
 using :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
 stream endpoint in each sub-device.
+
+Internal pads setup example
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A simple example of a multiplexed stream setup might be as follows:
+
+- A CCS camera sensor source sub-device, with one sink pad (0), one source pad

Explain what 'CCS' means: probably a link to MIPI CCS is sufficient, but for newbies 'CCS' doesn't provide any information.

Documentation/userspace-api/media/v4l/dev-subdev.rst A common way to accomplish this is to start from the sensors and propagate the · line 697

Quoted 1 line
Documentation/userspace-api/media/v4l/dev-subdev.rst +1 -0
+  (1), an internal sink pad (2) that represents the source of embedded

So since the INTERNAL flag is associated with a sink pad, this is now called an 'internal sink' instead of 'internal source' as it was before. I agree with that, but note that the phrase 'internal source' is still used in several places, including the cover letter. For the next version you post, please check for that and change the terminology since mixing the two is very confusing :-)

Quoted 28 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +28 -0
+  data. There are two routes, one from the sink pad to the source, and another
+  from the internal sink pad to the source pad. The embedded data stream needs
+  to be enabled by activating the related route. The configuration of the rest
+  of the CCS sub-devices is omitted from this example.
+
+- Multiplexer bridge (Bridge). The bridge has one sink pad, connected to the
+  sensor (pad 0), and one source pad (pad 1), which outputs two streams.
+
+- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
+  connected to the bridge, and two source pads (pads 1-2), going to the DMA
+  engine. The receiver demultiplexes the incoming streams to the source pads.
+
+- DMA Engines in the SoC (DMA Engine), one for each stream. Each DMA engine is
+  connected to a single source pad in the receiver.
+
+The sensor, the bridge and the receiver are modeled as V4L2 sub-devices,
+exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
+modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
+
+To configure this pipeline, the userspace must take the following steps:
+
+1) Set up media links between entities: connect the sensors to the bridge,
+   bridge to the receiver, and the receiver to the DMA engines. This step does
+   not differ from normal non-multiplexed media controller setup.
+
+2) Configure routing
+
+.. flat-table:: Camera sensor

Documentation/userspace-api/media/v4l/dev-subdev.rst A common way to accomplish this is to start from the sensors and propagate the · line 735

Quoted 10 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +10 -0
+    :header-rows: 1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from the sink pad

Huh? It is a sensor subdev, so aren't both pixel data and metadata coming from an internal sink?

I think you are actually describing the internal subdev of a CCS sensor here, but that is not clear from the description at all.

Since CCS sensors are relatively common (I think), it might be good to give a fully fledged example.

But suppose you have a non-CCS sensor that generates the pixel data and metadata, then you would have two internal sink pads and one source pad for two streams, right?

And very simple sensors that currently just have a single source pad could be describes as having an internal sink pad connected to the source pad by a fixed route. But we don't do that, because it is overkill. Correct? If so, then this might be useful information to add to answer the question why some sensors have internal sinks and others do not.

Quoted 102 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +102 -0
+    * - 2/0
+      - 1/1
+      - **V4L2_SUBDEV_ROUTE_FL_ACTIVE**
+      - Metadata stream from the internal sink pad
+
+.. flat-table:: Bridge routing table
+    :header-rows: 1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from camera sensor
+    * - 0/1
+      - 1/1
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from camera sensor
+
+.. flat-table:: Receiver routing table
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from camera sensor
+    * - 0/1
+      - 2/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from camera sensor
+
+The options available in sensor's routing configuration are dictated by hardware
+capabilities: typically camera sensors always produce image data stream while
+the embedded data stream typically can be either enabled or disabled.
+
+3) Configure formats and selections
+
+This example assumes that the formats are propagated from sink pad to the source
+pad as-is. The tables contain fields of both struct v4l2_subdev_format and
+struct v4l2_mbus_framefmt. The full configuration of CCS camera sensor is out of
+scope of this example.
+
+.. flat-table:: Formats set on the sub-devices. Bold values are set, others are
+                static or propagated.
+    :header-rows: 1
+    :fill-cells:
+
+    * - Sub-device
+      - Pad/Stream
+      - Width
+      - Height
+      - Code
+    * - :rspan:`3` Camera sensor sub-device (CCS source sub-device)
+      - 0/0
+      - 640
+      - 480
+      - MEDIA_BUS_FMT_SGRBG10
+    * - 1/0
+      - 640
+      - 480
+      - **MEDIA_BUS_FMT_SGRBG10**
+    * - 2/0
+      - 640
+      - 2
+      - MEDIA_BUS_FMT_CCS_EMBEDDED_10
+    * - 1/1
+      - 640
+      - 2
+      - MEDIA_BUS_FMT_META_10
+    * - :rspan:`3` Bridge
+      - 0/0
+      - **640**
+      - **480**
+      - **MEDIA_BUS_FMT_SGRBG10**
+    * - 1/0
+      - 640
+      - 480
+      - MEDIA_BUS_FMT_SGRBG10
+    * - 0/1
+      - **640**
+      - **2**
+      - **MEDIA_BUS_FMT_META_10**
+    * - 1/1
+      - 640
+      - 2
+      - MEDIA_BUS_FMT_META_10
+    * - :rspan:`3` Receiver
+      - 0/0
+      - **640**
+      - **480**
+      - **MEDIA_BUS_FMT_SGRBG10**
+    * - 1/0
+      - 640
+      - 480
+      - MEDIA_BUS_FMT_SGRBG10
+    * - 0/1

Documentation/userspace-api/media/v4l/dev-subdev.rst A common way to accomplish this is to start from the sensors and propagate the · line 847

Quoted 10 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +10 -0
+      - **640**
+      - **2**
+      - **MEDIA_BUS_FMT_META_10**
+    * - 2/0
+      - 640
+      - 2
+      - MEDIA_BUS_FMT_META_10
+
+The embedded data format does not need to be configured as the format is
+dictated by the pixel data format in this case.

Regards,

Hans

author maintainer
#81 lore raw

Re: [PATCH v3 10/10] media: uapi: v4l: Document source routes

Hi Hans,

Thank you for the review.

Quoted 28 lines · Hans Verkuil
Quoted 27 lines · Sakari Ailus

Document how internal pads are used on source routes.

---
 .../userspace-api/media/v4l/dev-subdev.rst    | 179 ++++++++++++++++++
 1 file changed, 179 insertions(+)
Documentation/userspace-api/media/v4l/dev-subdev.rst +12 -0
index b023918177b5..27b0fe2dc83a 100644
@@ -551,6 +551,27 @@ A stream at a specific point in the media pipeline is identified by the
 sub-device and a (pad, stream) pair. For sub-devices that do not support
 multiplexed streams the 'stream' field is always 0.
 
+.. _v4l2-subdev-source-routes:
+
+Internal pads and source routes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Cases where a single sub-device source pad is traversed by multiple streams one
+or more of which originate from within the sub-device itself are special as
+there is no external sink pad for such routes. In those cases, the sources of
+the internally generated streams are represented by internal sink pads, which
+are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`
+pad flag set.
+
Quoted 8 lines

Documentation/userspace-api/media/v4l/dev-subdev.rst A stream at a specific point in the media pipeline is identified by the · line 570

Quoted 5 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +5 -0
+Internal pads have all the properties of an external pad, including formats and
+selections. The format in this case is the source format of the stream. An
+internal pad always has a single stream only (0).
+
+/Source routes/ are routes from an internal sink pad to a(n external) source

'/Source routes/' appears just like that in the generated documentation, that's probably not what you intended.

This was intended to be *Source routes*. I'll address it in v4.

Quoted 23 lines
Quoted 9 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +4 -0
+pad. Generally source routes are not modifiable but they can be activated and
+deactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
+<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.
+
 Interaction between routes, streams, formats and selections
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

Documentation/userspace-api/media/v4l/dev-subdev.rst A common way to accomplish this is to start from the sensors and propagate the · line 696

Quoted 10 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +7 -0
@@ -666,3 +687,161 @@ A common way to accomplish this is to start from the sensors and propagate the
 configurations along the stream towards the receiver,
 using :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
 stream endpoint in each sub-device.
+
+Internal pads setup example
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A simple example of a multiplexed stream setup might be as follows:
+
+- A CCS camera sensor source sub-device, with one sink pad (0), one source pad

Explain what 'CCS' means: probably a link to MIPI CCS is sufficient, but for newbies 'CCS' doesn't provide any information.

Quoted 7 lines

Documentation/userspace-api/media/v4l/dev-subdev.rst A common way to accomplish this is to start from the sensors and propagate the · line 697

Quoted 1 line
Documentation/userspace-api/media/v4l/dev-subdev.rst +1 -0
+  (1), an internal sink pad (2) that represents the source of embedded

So since the INTERNAL flag is associated with a sink pad, this is now called an 'internal sink' instead of 'internal source' as it was before. I agree with that, but note that the phrase 'internal source' is still used in several places, including the cover letter. For the next version you post, please check for that and change the terminology since mixing the two is very confusing :-)

Are you referring to this patch? There don't seem to be any references to "intenal source pads" left.

Quoted 34 lines
Quoted 34 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +33 -0
+  data. There are two routes, one from the sink pad to the source, and another
+  from the internal sink pad to the source pad. The embedded data stream needs
+  to be enabled by activating the related route. The configuration of the rest
+  of the CCS sub-devices is omitted from this example.
+
+- Multiplexer bridge (Bridge). The bridge has one sink pad, connected to the
+  sensor (pad 0), and one source pad (pad 1), which outputs two streams.
+
+- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
+  connected to the bridge, and two source pads (pads 1-2), going to the DMA
+  engine. The receiver demultiplexes the incoming streams to the source pads.
+
+- DMA Engines in the SoC (DMA Engine), one for each stream. Each DMA engine is
+  connected to a single source pad in the receiver.
+
+The sensor, the bridge and the receiver are modeled as V4L2 sub-devices,
+exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
+modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
+
+To configure this pipeline, the userspace must take the following steps:
+
+1) Set up media links between entities: connect the sensors to the bridge,
+   bridge to the receiver, and the receiver to the DMA engines. This step does
+   not differ from normal non-multiplexed media controller setup.
+
+2) Configure routing
+
+.. flat-table:: Camera sensor
+    :header-rows: 1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
Quoted 8 lines

Documentation/userspace-api/media/v4l/dev-subdev.rst A common way to accomplish this is to start from the sensors and propagate the · line 735

Quoted 5 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +5 -0
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from the sink pad

Huh? It is a sensor subdev, so aren't both pixel data and metadata coming from an internal sink?

The CCS driver exposes three sub-devices, so in this case the sink pad isn't internal. Almost (?) all other sensor drivers have a single sub-device only, perhaps I'll change the example to align with one of those.

Quoted 130 lines

I think you are actually describing the internal subdev of a CCS sensor here, but that is not clear from the description at all.

Since CCS sensors are relatively common (I think), it might be good to give a fully fledged example.

But suppose you have a non-CCS sensor that generates the pixel data and metadata, then you would have two internal sink pads and one source pad for two streams, right?

And very simple sensors that currently just have a single source pad could be describes as having an internal sink pad connected to the source pad by a fixed route. But we don't do that, because it is overkill. Correct? If so, then this might be useful information to add to answer the question why some sensors have internal sinks and others do not.

Quoted 112 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +112 -0
+    * - 2/0
+      - 1/1
+      - **V4L2_SUBDEV_ROUTE_FL_ACTIVE**
+      - Metadata stream from the internal sink pad
+
+.. flat-table:: Bridge routing table
+    :header-rows: 1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from camera sensor
+    * - 0/1
+      - 1/1
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from camera sensor
+
+.. flat-table:: Receiver routing table
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from camera sensor
+    * - 0/1
+      - 2/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from camera sensor
+
+The options available in sensor's routing configuration are dictated by hardware
+capabilities: typically camera sensors always produce image data stream while
+the embedded data stream typically can be either enabled or disabled.
+
+3) Configure formats and selections
+
+This example assumes that the formats are propagated from sink pad to the source
+pad as-is. The tables contain fields of both struct v4l2_subdev_format and
+struct v4l2_mbus_framefmt. The full configuration of CCS camera sensor is out of
+scope of this example.
+
+.. flat-table:: Formats set on the sub-devices. Bold values are set, others are
+                static or propagated.
+    :header-rows: 1
+    :fill-cells:
+
+    * - Sub-device
+      - Pad/Stream
+      - Width
+      - Height
+      - Code
+    * - :rspan:`3` Camera sensor sub-device (CCS source sub-device)
+      - 0/0
+      - 640
+      - 480
+      - MEDIA_BUS_FMT_SGRBG10
+    * - 1/0
+      - 640
+      - 480
+      - **MEDIA_BUS_FMT_SGRBG10**
+    * - 2/0
+      - 640
+      - 2
+      - MEDIA_BUS_FMT_CCS_EMBEDDED_10
+    * - 1/1
+      - 640
+      - 2
+      - MEDIA_BUS_FMT_META_10
+    * - :rspan:`3` Bridge
+      - 0/0
+      - **640**
+      - **480**
+      - **MEDIA_BUS_FMT_SGRBG10**
+    * - 1/0
+      - 640
+      - 480
+      - MEDIA_BUS_FMT_SGRBG10
+    * - 0/1
+      - **640**
+      - **2**
+      - **MEDIA_BUS_FMT_META_10**
+    * - 1/1
+      - 640
+      - 2
+      - MEDIA_BUS_FMT_META_10
+    * - :rspan:`3` Receiver
+      - 0/0
+      - **640**
+      - **480**
+      - **MEDIA_BUS_FMT_SGRBG10**
+    * - 1/0
+      - 640
+      - 480
+      - MEDIA_BUS_FMT_SGRBG10
+    * - 0/1
+      - **640**
+      - **2**
+      - **MEDIA_BUS_FMT_META_10**
+    * - 2/0
+      - 640
+      - 2
+      - MEDIA_BUS_FMT_META_10
+
+The embedded data format does not need to be configured as the format is
+dictated by the pixel data format in this case.
Signature · 3 lines
Kind regards,

Sakari Ailus
maintainer
#82 lore raw

Re: [PATCH v3 10/10] media: uapi: v4l: Document source routes

Hi Sakari,

Thank you for the patch.

Quoted 26 lines · Sakari Ailus
Quoted 25 lines · Hans Verkuil
Quoted 24 lines · Sakari Ailus

Document how internal pads are used on source routes.

---
 .../userspace-api/media/v4l/dev-subdev.rst    | 179 ++++++++++++++++++
 1 file changed, 179 insertions(+)
Documentation/userspace-api/media/v4l/dev-subdev.rst +9 -0
index b023918177b5..27b0fe2dc83a 100644
@@ -551,6 +551,27 @@ A stream at a specific point in the media pipeline is identified by the
 sub-device and a (pad, stream) pair. For sub-devices that do not support
 multiplexed streams the 'stream' field is always 0.
 
+.. _v4l2-subdev-source-routes:
+
+Internal pads and source routes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Cases where a single sub-device source pad is traversed by multiple streams one
+or more of which originate from within the sub-device itself are special as
+there is no external sink pad for such routes. In those cases, the sources of
+the internally generated streams are represented by internal sink pads, which
Quoted 8 lines
Quoted 8 lines

Documentation/userspace-api/media/v4l/dev-subdev.rst A stream at a specific point in the media pipeline is identified by the · line 570

Quoted 8 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +8 -0
+are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`
+pad flag set.
+
+Internal pads have all the properties of an external pad, including formats and
+selections. The format in this case is the source format of the stream. An
+internal pad always has a single stream only (0).
+
+/Source routes/ are routes from an internal sink pad to a(n external) source

"a(n external)" looks weird.

Quoted 8 lines
Quoted 2 lines

'/Source routes/' appears just like that in the generated documentation, that's probably not what you intended.

This was intended to be *Source routes*. I'll address it in v4.

Quoted 3 lines

Documentation/userspace-api/media/v4l/dev-subdev.rst A stream at a specific point in the media pipeline is identified by the · line 573

Quoted 3 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +3 -0
+pad. Generally source routes are not modifiable but they can be activated and
+deactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
+<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.

We need to update the documentation of the routing API to clearly explain the different models (i.e. specifying all routes, active and inactive, vs. including only active routes). That's out of scope for this patch, but it could be done in the same series.

Quoted 21 lines
Quoted 21 lines
Quoted 5 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +1 -0
+
 Interaction between routes, streams, formats and selections
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

Documentation/userspace-api/media/v4l/dev-subdev.rst A common way to accomplish this is to start from the sensors and propagate the · line 696

Quoted 10 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +7 -0
@@ -666,3 +687,161 @@ A common way to accomplish this is to start from the sensors and propagate the
 configurations along the stream towards the receiver,
 using :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
 stream endpoint in each sub-device.
+
+Internal pads setup example
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A simple example of a multiplexed stream setup might be as follows:
+
+- A CCS camera sensor source sub-device, with one sink pad (0), one source pad

Explain what 'CCS' means: probably a link to MIPI CCS is sufficient, but for newbies 'CCS' doesn't provide any information.

Quoted 1 line
Documentation/userspace-api/media/v4l/dev-subdev.rst +1 -0
+  (1), an internal sink pad (2) that represents the source of embedded
Quoted 13 lines
Quoted 5 lines

So since the INTERNAL flag is associated with a sink pad, this is now called an 'internal sink' instead of 'internal source' as it was before. I agree with that, but note that the phrase 'internal source' is still used in several places, including the cover letter. For the next version you post, please check for that and change the terminology since mixing the two is very confusing :-)

Are you referring to this patch? There don't seem to be any references to "intenal source pads" left.

Quoted 4 lines
Quoted 4 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +4 -0
+  data. There are two routes, one from the sink pad to the source, and another
+  from the internal sink pad to the source pad. The embedded data stream needs
+  to be enabled by activating the related route. The configuration of the rest
+  of the CCS sub-devices is omitted from this example.

CCS is complex, with its multiple subdevs. I think a simpler raw sensor without a scaler, using a single subdev, would be a better example.

Quoted 38 lines
Quoted 29 lines
Quoted 29 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +29 -0
+
+- Multiplexer bridge (Bridge). The bridge has one sink pad, connected to the
+  sensor (pad 0), and one source pad (pad 1), which outputs two streams.
+
+- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
+  connected to the bridge, and two source pads (pads 1-2), going to the DMA
+  engine. The receiver demultiplexes the incoming streams to the source pads.
+
+- DMA Engines in the SoC (DMA Engine), one for each stream. Each DMA engine is
+  connected to a single source pad in the receiver.
+
+The sensor, the bridge and the receiver are modeled as V4L2 sub-devices,
+exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
+modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
+
+To configure this pipeline, the userspace must take the following steps:
+
+1) Set up media links between entities: connect the sensors to the bridge,
+   bridge to the receiver, and the receiver to the DMA engines. This step does
+   not differ from normal non-multiplexed media controller setup.
+
+2) Configure routing
+
+.. flat-table:: Camera sensor
+    :header-rows: 1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
Quoted 8 lines

Documentation/userspace-api/media/v4l/dev-subdev.rst A common way to accomplish this is to start from the sensors and propagate the · line 735

Quoted 5 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +5 -0
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from the sink pad

Huh? It is a sensor subdev, so aren't both pixel data and metadata coming from an internal sink?

Quoted 4 lines

The CCS driver exposes three sub-devices, so in this case the sink pad isn't internal. Almost (?) all other sensor drivers have a single sub-device only, perhaps I'll change the example to align with one of those.

Ah, it looks like we have thought about the same thing :-)

Quoted 129 lines
Quoted 129 lines

I think you are actually describing the internal subdev of a CCS sensor here, but that is not clear from the description at all.

Since CCS sensors are relatively common (I think), it might be good to give a fully fledged example.

But suppose you have a non-CCS sensor that generates the pixel data and metadata, then you would have two internal sink pads and one source pad for two streams, right?

And very simple sensors that currently just have a single source pad could be describes as having an internal sink pad connected to the source pad by a fixed route. But we don't do that, because it is overkill. Correct? If so, then this might be useful information to add to answer the question why some sensors have internal sinks and others do not.

Quoted 112 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +112 -0
+    * - 2/0
+      - 1/1
+      - **V4L2_SUBDEV_ROUTE_FL_ACTIVE**
+      - Metadata stream from the internal sink pad
+
+.. flat-table:: Bridge routing table
+    :header-rows: 1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from camera sensor
+    * - 0/1
+      - 1/1
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from camera sensor
+
+.. flat-table:: Receiver routing table
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from camera sensor
+    * - 0/1
+      - 2/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from camera sensor
+
+The options available in sensor's routing configuration are dictated by hardware
+capabilities: typically camera sensors always produce image data stream while
+the embedded data stream typically can be either enabled or disabled.
+
+3) Configure formats and selections
+
+This example assumes that the formats are propagated from sink pad to the source
+pad as-is. The tables contain fields of both struct v4l2_subdev_format and
+struct v4l2_mbus_framefmt. The full configuration of CCS camera sensor is out of
+scope of this example.
+
+.. flat-table:: Formats set on the sub-devices. Bold values are set, others are
+                static or propagated.
+    :header-rows: 1
+    :fill-cells:
+
+    * - Sub-device
+      - Pad/Stream
+      - Width
+      - Height
+      - Code
+    * - :rspan:`3` Camera sensor sub-device (CCS source sub-device)
+      - 0/0
+      - 640
+      - 480
+      - MEDIA_BUS_FMT_SGRBG10
+    * - 1/0
+      - 640
+      - 480
+      - **MEDIA_BUS_FMT_SGRBG10**
+    * - 2/0
+      - 640
+      - 2
+      - MEDIA_BUS_FMT_CCS_EMBEDDED_10
+    * - 1/1
+      - 640
+      - 2
+      - MEDIA_BUS_FMT_META_10
+    * - :rspan:`3` Bridge
+      - 0/0
+      - **640**
+      - **480**
+      - **MEDIA_BUS_FMT_SGRBG10**
+    * - 1/0
+      - 640
+      - 480
+      - MEDIA_BUS_FMT_SGRBG10
+    * - 0/1
+      - **640**
+      - **2**
+      - **MEDIA_BUS_FMT_META_10**
+    * - 1/1
+      - 640
+      - 2
+      - MEDIA_BUS_FMT_META_10
+    * - :rspan:`3` Receiver
+      - 0/0
+      - **640**
+      - **480**
+      - **MEDIA_BUS_FMT_SGRBG10**
+    * - 1/0
+      - 640
+      - 480
+      - MEDIA_BUS_FMT_SGRBG10
+    * - 0/1
+      - **640**
+      - **2**
+      - **MEDIA_BUS_FMT_META_10**
+    * - 2/0
+      - 640
+      - 2
+      - MEDIA_BUS_FMT_META_10
+
+The embedded data format does not need to be configured as the format is
+dictated by the pixel data format in this case.
Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#83 lore raw

Re: [PATCH v3 10/10] media: uapi: v4l: Document source routes

Hi Laurent,

Thank you for the review.

Quoted 33 lines · Laurent Pinchart

Hi Sakari,

Thank you for the patch.

Quoted 28 lines · Sakari Ailus
Quoted 27 lines · Hans Verkuil
Quoted 26 lines · Sakari Ailus

Document how internal pads are used on source routes.

---
 .../userspace-api/media/v4l/dev-subdev.rst    | 179 ++++++++++++++++++
 1 file changed, 179 insertions(+)
Documentation/userspace-api/media/v4l/dev-subdev.rst +11 -0
index b023918177b5..27b0fe2dc83a 100644
@@ -551,6 +551,27 @@ A stream at a specific point in the media pipeline is identified by the
 sub-device and a (pad, stream) pair. For sub-devices that do not support
 multiplexed streams the 'stream' field is always 0.
 
+.. _v4l2-subdev-source-routes:
+
+Internal pads and source routes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Cases where a single sub-device source pad is traversed by multiple streams one
+or more of which originate from within the sub-device itself are special as
+there is no external sink pad for such routes. In those cases, the sources of
+the internally generated streams are represented by internal sink pads, which
+are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`
+pad flag set.
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines

Documentation/userspace-api/media/v4l/dev-subdev.rst A stream at a specific point in the media pipeline is identified by the · line 570

Quoted 6 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +6 -0
+
+Internal pads have all the properties of an external pad, including formats and
+selections. The format in this case is the source format of the stream. An
+internal pad always has a single stream only (0).
+
+/Source routes/ are routes from an internal sink pad to a(n external) source

"a(n external)" looks weird.

I'll remove the parentheses. Both are applicable. We don't generally call pads external but saying that here is relevant.

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

'/Source routes/' appears just like that in the generated documentation, that's probably not what you intended.

This was intended to be *Source routes*. I'll address it in v4.

Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

Documentation/userspace-api/media/v4l/dev-subdev.rst A stream at a specific point in the media pipeline is identified by the · line 573

Quoted 3 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +3 -0
+pad. Generally source routes are not modifiable but they can be activated and
+deactivated using the :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE
+<v4l2-subdev-routing-flags>` flag, depending on driver capabilities.

We need to update the documentation of the routing API to clearly explain the different models (i.e. specifying all routes, active and inactive, vs. including only active routes). That's out of scope for this patch, but it could be done in the same series.

I agree.

With this series eventually merged, I'm also hoping we could enable the streams API.

Quoted 213 lines
Quoted 22 lines
Quoted 22 lines
Quoted 6 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +1 -0
+
 Interaction between routes, streams, formats and selections
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

Documentation/userspace-api/media/v4l/dev-subdev.rst A common way to accomplish this is to start from the sensors and propagate the · line 696

Quoted 10 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +7 -0
@@ -666,3 +687,161 @@ A common way to accomplish this is to start from the sensors and propagate the
 configurations along the stream towards the receiver,
 using :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
 stream endpoint in each sub-device.
+
+Internal pads setup example
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A simple example of a multiplexed stream setup might be as follows:
+
+- A CCS camera sensor source sub-device, with one sink pad (0), one source pad

Explain what 'CCS' means: probably a link to MIPI CCS is sufficient, but for newbies 'CCS' doesn't provide any information.

Quoted 1 line
Documentation/userspace-api/media/v4l/dev-subdev.rst +1 -0
+  (1), an internal sink pad (2) that represents the source of embedded
Quoted 13 lines
Quoted 5 lines

So since the INTERNAL flag is associated with a sink pad, this is now called an 'internal sink' instead of 'internal source' as it was before. I agree with that, but note that the phrase 'internal source' is still used in several places, including the cover letter. For the next version you post, please check for that and change the terminology since mixing the two is very confusing :-)

Are you referring to this patch? There don't seem to be any references to "intenal source pads" left.

Quoted 4 lines
Quoted 4 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +4 -0
+  data. There are two routes, one from the sink pad to the source, and another
+  from the internal sink pad to the source pad. The embedded data stream needs
+  to be enabled by activating the related route. The configuration of the rest
+  of the CCS sub-devices is omitted from this example.

CCS is complex, with its multiple subdevs. I think a simpler raw sensor without a scaler, using a single subdev, would be a better example.

Quoted 38 lines
Quoted 29 lines
Quoted 29 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +29 -0
+
+- Multiplexer bridge (Bridge). The bridge has one sink pad, connected to the
+  sensor (pad 0), and one source pad (pad 1), which outputs two streams.
+
+- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
+  connected to the bridge, and two source pads (pads 1-2), going to the DMA
+  engine. The receiver demultiplexes the incoming streams to the source pads.
+
+- DMA Engines in the SoC (DMA Engine), one for each stream. Each DMA engine is
+  connected to a single source pad in the receiver.
+
+The sensor, the bridge and the receiver are modeled as V4L2 sub-devices,
+exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
+modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
+
+To configure this pipeline, the userspace must take the following steps:
+
+1) Set up media links between entities: connect the sensors to the bridge,
+   bridge to the receiver, and the receiver to the DMA engines. This step does
+   not differ from normal non-multiplexed media controller setup.
+
+2) Configure routing
+
+.. flat-table:: Camera sensor
+    :header-rows: 1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
Quoted 8 lines

Documentation/userspace-api/media/v4l/dev-subdev.rst A common way to accomplish this is to start from the sensors and propagate the · line 735

Quoted 5 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +5 -0
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from the sink pad

Huh? It is a sensor subdev, so aren't both pixel data and metadata coming from an internal sink?

Quoted 4 lines

The CCS driver exposes three sub-devices, so in this case the sink pad isn't internal. Almost (?) all other sensor drivers have a single sub-device only, perhaps I'll change the example to align with one of those.

Ah, it looks like we have thought about the same thing :-)

Quoted 129 lines
Quoted 129 lines

I think you are actually describing the internal subdev of a CCS sensor here, but that is not clear from the description at all.

Since CCS sensors are relatively common (I think), it might be good to give a fully fledged example.

But suppose you have a non-CCS sensor that generates the pixel data and metadata, then you would have two internal sink pads and one source pad for two streams, right?

And very simple sensors that currently just have a single source pad could be describes as having an internal sink pad connected to the source pad by a fixed route. But we don't do that, because it is overkill. Correct? If so, then this might be useful information to add to answer the question why some sensors have internal sinks and others do not.

Quoted 112 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +112 -0
+    * - 2/0
+      - 1/1
+      - **V4L2_SUBDEV_ROUTE_FL_ACTIVE**
+      - Metadata stream from the internal sink pad
+
+.. flat-table:: Bridge routing table
+    :header-rows: 1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from camera sensor
+    * - 0/1
+      - 1/1
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from camera sensor
+
+.. flat-table:: Receiver routing table
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from camera sensor
+    * - 0/1
+      - 2/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from camera sensor
+
+The options available in sensor's routing configuration are dictated by hardware
+capabilities: typically camera sensors always produce image data stream while
+the embedded data stream typically can be either enabled or disabled.
+
+3) Configure formats and selections
+
+This example assumes that the formats are propagated from sink pad to the source
+pad as-is. The tables contain fields of both struct v4l2_subdev_format and
+struct v4l2_mbus_framefmt. The full configuration of CCS camera sensor is out of
+scope of this example.
+
+.. flat-table:: Formats set on the sub-devices. Bold values are set, others are
+                static or propagated.
+    :header-rows: 1
+    :fill-cells:
+
+    * - Sub-device
+      - Pad/Stream
+      - Width
+      - Height
+      - Code
+    * - :rspan:`3` Camera sensor sub-device (CCS source sub-device)
+      - 0/0
+      - 640
+      - 480
+      - MEDIA_BUS_FMT_SGRBG10
+    * - 1/0
+      - 640
+      - 480
+      - **MEDIA_BUS_FMT_SGRBG10**
+    * - 2/0
+      - 640
+      - 2
+      - MEDIA_BUS_FMT_CCS_EMBEDDED_10
+    * - 1/1
+      - 640
+      - 2
+      - MEDIA_BUS_FMT_META_10
+    * - :rspan:`3` Bridge
+      - 0/0
+      - **640**
+      - **480**
+      - **MEDIA_BUS_FMT_SGRBG10**
+    * - 1/0
+      - 640
+      - 480
+      - MEDIA_BUS_FMT_SGRBG10
+    * - 0/1
+      - **640**
+      - **2**
+      - **MEDIA_BUS_FMT_META_10**
+    * - 1/1
+      - 640
+      - 2
+      - MEDIA_BUS_FMT_META_10
+    * - :rspan:`3` Receiver
+      - 0/0
+      - **640**
+      - **480**
+      - **MEDIA_BUS_FMT_SGRBG10**
+    * - 1/0
+      - 640
+      - 480
+      - MEDIA_BUS_FMT_SGRBG10
+    * - 0/1
+      - **640**
+      - **2**
+      - **MEDIA_BUS_FMT_META_10**
+    * - 2/0
+      - 640
+      - 2
+      - MEDIA_BUS_FMT_META_10
+
+The embedded data format does not need to be configured as the format is
+dictated by the pixel data format in this case.
Signature · 3 lines
Regards,

Sakari Ailus