media: Documentation: Interaction between routes, formats and selections
PATCH v2 1/1 media: Documentation: Interaction between routes, formats and selections
Document how setting up routes interacts with formats and selections.
---
since v1:
- Rewording changes based on Tomi's comments. In particular stricter separation between "stream" and "stream ID".
.../userspace-api/media/v4l/dev-subdev.rst | 50 ++++++++++++++----- 1 file changed, 37 insertions(+), 13 deletions(-)
Documentation/userspace-api/media/v4l/dev-subdev.rst +37 -13
@@ -406,6 +406,8 @@ pixel array is not rectangular but cross-shaped or round. The maximum size may also be smaller than the BOUNDS rectangle. +.. _format-propagation: + Order of configuration and format propagation ---------------------------------------------
@@ -507,12 +509,12 @@ source pads. Streams, multiplexed media pads and internal routing ---------------------------------------------------- -Commonly V4L2 subdevices support only separate video streams, that is, only a -single stream can pass through a media link and a media pad. Thus each pad -contains a format configuration for that single stream. In some cases a subdev -can do stream processing and split a stream into two or compose two streams -into one, but the inputs and outputs for the subdev are still a single stream -per pad. +Simple V4L2 subdevices do not support multiple, unrelated video streams, +and only a single stream can pass through a media link and a media pad. +Thus each pad contains a format and selection configuration for that +single stream. A subdev can do stream processing and split a stream into +two or compose two streams into one, but the inputs and outputs for the +subdev are still a single stream per pad. Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is, multiple data streams are transmitted on the same bus, which is represented by a media
@@ -539,14 +541,33 @@ streams from one end of the link to the other, and subdevices have routing tables which describe how the incoming streams from sink pads are routed to the source pads. -A stream ID (often just "stream") is a media link-local identifier for a stream. +A stream ID is a media link-local identifier for a stream. In other words, a particular stream ID must exist on both sides of a media link, but another stream ID can be used for the same stream at the other side -of the subdevice. +of the subdevice. The same stream ID is used to refer to the stream on +both pads of the link on all ioctls operating on pads. + +A stream at a specific point in the media pipeline is identified by the +sub-devdev and a pad ID-stream ID pair. For subdevices that do not support +multiplexed streams the 'stream' field is always 0. + +Interaction between routes, formats and selections +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The addition of routes to the V4L2 sub-device interface moves the +sub-device formats and selections from pads to pad ID-stream ID pairs. +Besides the usual pad ID, also the stream ID needs to be provided for +setting formats and selections. + +Instead of the sub-device wide merging of streams from all source pads +towards all sink pads, data flows for each route are separate from each +other. Any number of routes from streams on sink pads towards streams on +source pads is allowed, to the extent supported by drivers. For every +stream on a sink pad, however, only a single route is allowed. -A stream at a specific point in the media pipeline is identified with the -subdev and a (pad, stream) pair. For subdevices that do not support -multiplexed streams the 'stream' is always 0. +Stream IDs, part of routes, are configured by using the +:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. This will +also revert the format and selection configuration to device defaults. Configuring streams ^^^^^^^^^^^^^^^^^^^
@@ -565,8 +586,11 @@ Controller API <media_controller>` setting the routing table will reset all the stream configurations in a media entity. -3) Configure streams. Each route endpoint must be configured -with :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>`. +3) Configure formats and selections. Each route is configured separately +as documented plain subdevices in :ref:`<format-propagation>`. The stream +ID is set to the same stream ID associated with either sink or source pads +of routes configured using the :ref:`VIDIOC_SUBDEV_S_ROUTING +<VIDIOC_SUBDEV_G_ROUTING>` ioctl. Multiplexed streams setup example ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Signature · 1 line
2.30.2
Hi Sakari,
Thank you for the patch.
Quoted 51 lines · Sakari Ailus
Document how setting up routes interacts with formats and selections.
Signed-off-by Sakari Ailus <[email protected]>---
since v1:- Rewording changes based on Tomi's comments. In particular stricter separation between "stream" and "stream ID".
.../userspace-api/media/v4l/dev-subdev.rst | 50 ++++++++++++++----- 1 file changed, 37 insertions(+), 13 deletions(-)Documentation/userspace-api/media/v4l/dev-subdev.rst +9
-7@@ -406,6 +406,8 @@ pixel array is not rectangular but cross-shaped or round. The maximum size may also be smaller than the BOUNDS rectangle. +.. _format-propagation: + Order of configuration and format propagation ---------------------------------------------@@ -507,12 +509,12 @@ source pads. Streams, multiplexed media pads and internal routing ---------------------------------------------------- -Commonly V4L2 subdevices support only separate video streams, that is, only a -single stream can pass through a media link and a media pad. Thus each pad -contains a format configuration for that single stream. In some cases a subdev -can do stream processing and split a stream into two or compose two streams -into one, but the inputs and outputs for the subdev are still a single stream -per pad. +Simple V4L2 subdevices do not support multiple, unrelated video streams, +and only a single stream can pass through a media link and a media pad. +Thus each pad contains a format and selection configuration for that +single stream. A subdev can do stream processing and split a stream into +two or compose two streams into one, but the inputs and outputs for the +subdev are still a single stream per pad. Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is, multiple data streams are transmitted on the same bus, which is represented by a media@@ -539,14 +541,33 @@ streams from one end of the link to the other, and subdevices have routing tables which describe how the incoming streams from sink pads are routed to the source pads. -A stream ID (often just "stream") is a media link-local identifier for a stream. +A stream ID is a media link-local identifier for a stream. In other words, a particular stream ID must exist on both sides of a media
Documentation/userspace-api/media/v4l/dev-subdev.rst streams from one end of the link to the other, and subdevices have routing · line 554
Quoted 10 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +8
-1link, but another stream ID can be used for the same stream at the other side -of the subdevice. +of the subdevice. The same stream ID is used to refer to the stream on +both pads of the link on all ioctls operating on pads. + +A stream at a specific point in the media pipeline is identified by the +sub-devdev and a pad ID-stream ID pair. For subdevices that do not support +multiplexed streams the 'stream' field is always 0. + +Interaction between routes, formats and selections
s/selections/selection rectangles/
Documentation/userspace-api/media/v4l/dev-subdev.rst streams from one end of the link to the other, and subdevices have routing · line 558
Quoted 4 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +4
-0+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The addition of routes to the V4L2 sub-device interface moves the +sub-device formats and selections from pads to pad ID-stream ID pairs.
This reads as "pad" "ID-stream" "ID" "pairs". Even knowing how the API behaves, I had trouble understanding this.
Documentation/userspace-api/media/v4l/dev-subdev.rst streams from one end of the link to the other, and subdevices have routing · line 566
Quoted 8 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +8
-0+Besides the usual pad ID, also the stream ID needs to be provided for +setting formats and selections. + +Instead of the sub-device wide merging of streams from all source pads +towards all sink pads, data flows for each route are separate from each +other. Any number of routes from streams on sink pads towards streams on +source pads is allowed, to the extent supported by drivers. For every +stream on a sink pad, however, only a single route is allowed.
This explains routes, not format and selection rectangles.
Documentation/userspace-api/media/v4l/dev-subdev.rst streams from one end of the link to the other, and subdevices have routing · line 570
Quoted 6 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +3
-3-A stream at a specific point in the media pipeline is identified with the -subdev and a (pad, stream) pair. For subdevices that do not support -multiplexed streams the 'stream' is always 0. +Stream IDs, part of routes, are configured by using the +:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. This will +also revert the format and selection configuration to device defaults.
I don't expect someone who doesn't have detailed knowledge of the implementation to understand what you mean here :-(
Documentation/userspace-api/media/v4l/dev-subdev.rst Controller API <media_controller>` · line 589
Quoted 9 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +1
-2Configuring streams ^^^^^^^^^^^^^^^^^^^@@ -565,8 +586,11 @@ Controller API <media_controller>` setting the routing table will reset all the stream configurations in a media entity. -3) Configure streams. Each route endpoint must be configured -with :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>`. +3) Configure formats and selections. Each route is configured separately
We don't configure routes but streams, don't we ? Mentioning routes here seems confusing to me.
Documentation/userspace-api/media/v4l/dev-subdev.rst Controller API <media_controller>` · line 590
Quoted 1 line
Documentation/userspace-api/media/v4l/dev-subdev.rst +1
-0+as documented plain subdevices in :ref:`<format-propagation>`. The stream
Did you mean "as documented for plain subdevices" ?
This is the part that bothers me. I don't think we can consider the configuration of routes to be independent in the general case. How the formats and selection rectangle of different streams interact with each other is what I'd like to see documented.
As this is an area that has been entirely unexplored until now as far as I can tell, I would be fine if we wanted to merge streams support without support for selection rectangles to start with, and experiment with the selection API later, once we have good use cases that can be used for design and development.
Quoted 6 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +3
-0+ID is set to the same stream ID associated with either sink or source pads +of routes configured using the :ref:`VIDIOC_SUBDEV_S_ROUTING +<VIDIOC_SUBDEV_G_ROUTING>` ioctl. Multiplexed streams setup example ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Signature · 3 lines
Regards, Laurent Pinchart
Hi Laurent,
Quoted 3 lines · Laurent Pinchart
Hi Sakari,
Thank you for the patch.
Thanks for the review!
Quoted 66 lines · Sakari Ailus
Quoted 51 lines
Document how setting up routes interacts with formats and selections.
Signed-off-by Sakari Ailus <[email protected]>---
since v1:- Rewording changes based on Tomi's comments. In particular stricter separation between "stream" and "stream ID".
.../userspace-api/media/v4l/dev-subdev.rst | 50 ++++++++++++++----- 1 file changed, 37 insertions(+), 13 deletions(-)Documentation/userspace-api/media/v4l/dev-subdev.rst +9
-7@@ -406,6 +406,8 @@ pixel array is not rectangular but cross-shaped or round. The maximum size may also be smaller than the BOUNDS rectangle. +.. _format-propagation: + Order of configuration and format propagation ---------------------------------------------@@ -507,12 +509,12 @@ source pads. Streams, multiplexed media pads and internal routing ---------------------------------------------------- -Commonly V4L2 subdevices support only separate video streams, that is, only a -single stream can pass through a media link and a media pad. Thus each pad -contains a format configuration for that single stream. In some cases a subdev -can do stream processing and split a stream into two or compose two streams -into one, but the inputs and outputs for the subdev are still a single stream -per pad. +Simple V4L2 subdevices do not support multiple, unrelated video streams, +and only a single stream can pass through a media link and a media pad. +Thus each pad contains a format and selection configuration for that +single stream. A subdev can do stream processing and split a stream into +two or compose two streams into one, but the inputs and outputs for the +subdev are still a single stream per pad. Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is, multiple data streams are transmitted on the same bus, which is represented by a media@@ -539,14 +541,33 @@ streams from one end of the link to the other, and subdevices have routing tables which describe how the incoming streams from sink pads are routed to the source pads. -A stream ID (often just "stream") is a media link-local identifier for a stream. +A stream ID is a media link-local identifier for a stream. In other words, a particular stream ID must exist on both sides of a mediaDocumentation/userspace-api/media/v4l/dev-subdev.rst streams from one end of the link to the other, and subdevices have routing · line 554
Quoted 10 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +8
-1link, but another stream ID can be used for the same stream at the other side -of the subdevice. +of the subdevice. The same stream ID is used to refer to the stream on +both pads of the link on all ioctls operating on pads. + +A stream at a specific point in the media pipeline is identified by the +sub-devdev and a pad ID-stream ID pair. For subdevices that do not support +multiplexed streams the 'stream' field is always 0. + +Interaction between routes, formats and selectionss/selections/selection rectangles/
Quoted 7 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst streams from one end of the link to the other, and subdevices have routing · line 558
Quoted 4 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +4
-0+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The addition of routes to the V4L2 sub-device interface moves the +sub-device formats and selections from pads to pad ID-stream ID pairs.This reads as "pad" "ID-stream" "ID" "pairs". Even knowing how the API behaves, I had trouble understanding this.
Well, that's perhaps a bit exaggerated but I agree the readability could be improved.
Maybe:
(pad ID, stream ID) pair
? Not exactly great perhaps but should be easier to read.
Quoted 11 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst streams from one end of the link to the other, and subdevices have routing · line 566
Quoted 9 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +8
-0+Besides the usual pad ID, also the stream ID needs to be provided for +setting formats and selections. + +Instead of the sub-device wide merging of streams from all source pads +towards all sink pads, data flows for each route are separate from each +other. Any number of routes from streams on sink pads towards streams on +source pads is allowed, to the extent supported by drivers. For every +stream on a sink pad, however, only a single route is allowed.This explains routes, not format and selection rectangles.
Perhaps a few more words on formats and selections would be appropriate indeed.
Quoted 17 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst streams from one end of the link to the other, and subdevices have routing · line 570
Quoted 7 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +3
-3-A stream at a specific point in the media pipeline is identified with the -subdev and a (pad, stream) pair. For subdevices that do not support -multiplexed streams the 'stream' is always 0. +Stream IDs, part of routes, are configured by using the +:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. This will +also revert the format and selection configuration to device defaults.I don't expect someone who doesn't have detailed knowledge of the implementation to understand what you mean here :-(
Quoted 6 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +0
-0Configuring streams ^^^^^^^^^^^^^^^^^^^@@ -565,8 +586,11 @@ Controller API <media_controller>` setting the routing table will reset all the stream configurations in a media entity.
Quoted 6 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst Controller API <media_controller>` · line 589
Quoted 3 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +1
-2-3) Configure streams. Each route endpoint must be configured -with :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>`. +3) Configure formats and selections. Each route is configured separatelyWe don't configure routes but streams, don't we ? Mentioning routes here seems confusing to me.
Well, routing, format and selection APIs all apply to streams, don't they?
I'll see if I can make this easier to grasp.
Quoted 4 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst Controller API <media_controller>` · line 590
Quoted 2 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +1
-0+as documented plain subdevices in :ref:`<format-propagation>`. The streamDid you mean "as documented for plain subdevices" ?
Yes.
Quoted 5 lines
This is the part that bothers me. I don't think we can consider the configuration of routes to be independent in the general case. How the formats and selection rectangle of different streams interact with each other is what I'd like to see documented.
That is device specific in the end. Even without routes there are hardware specific limitations on what can be done, in this case it's just that other routes can be limiting as well.
But as both are configured by the user, what takes priority, the pre-existing configuration or the new configuration set by the user? I think we indeed need rules for that.
Quoted 6 lines
As this is an area that has been entirely unexplored until now as far as I can tell, I would be fine if we wanted to merge streams support without support for selection rectangles to start with, and experiment with the selection API later, once we have good use cases that can be used for design and development.
Formats have dependencies, too, but perhaps a little less than if you add selections to the mix, too. In any case this should be documented in a different section.
Quoted 8 lines
Quoted 7 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +3
-0+ID is set to the same stream ID associated with either sink or source pads +of routes configured using the :ref:`VIDIOC_SUBDEV_S_ROUTING +<VIDIOC_SUBDEV_G_ROUTING>` ioctl. Multiplexed streams setup example ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Signature · 3 lines
Kind regards, Sakari Ailus
Hi Sakari,
Quoted 74 lines · Sakari Ailus
Quoted 65 lines · Laurent Pinchart
Quoted 51 lines · Sakari Ailus
Document how setting up routes interacts with formats and selections.
Signed-off-by Sakari Ailus <[email protected]>---
since v1:- Rewording changes based on Tomi's comments. In particular stricter separation between "stream" and "stream ID".
.../userspace-api/media/v4l/dev-subdev.rst | 50 ++++++++++++++----- 1 file changed, 37 insertions(+), 13 deletions(-)Documentation/userspace-api/media/v4l/dev-subdev.rst +9
-7@@ -406,6 +406,8 @@ pixel array is not rectangular but cross-shaped or round. The maximum size may also be smaller than the BOUNDS rectangle. +.. _format-propagation: + Order of configuration and format propagation ---------------------------------------------@@ -507,12 +509,12 @@ source pads. Streams, multiplexed media pads and internal routing ---------------------------------------------------- -Commonly V4L2 subdevices support only separate video streams, that is, only a -single stream can pass through a media link and a media pad. Thus each pad -contains a format configuration for that single stream. In some cases a subdev -can do stream processing and split a stream into two or compose two streams -into one, but the inputs and outputs for the subdev are still a single stream -per pad. +Simple V4L2 subdevices do not support multiple, unrelated video streams, +and only a single stream can pass through a media link and a media pad. +Thus each pad contains a format and selection configuration for that +single stream. A subdev can do stream processing and split a stream into +two or compose two streams into one, but the inputs and outputs for the +subdev are still a single stream per pad. Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is, multiple data streams are transmitted on the same bus, which is represented by a media@@ -539,14 +541,33 @@ streams from one end of the link to the other, and subdevices have routing tables which describe how the incoming streams from sink pads are routed to the source pads. -A stream ID (often just "stream") is a media link-local identifier for a stream. +A stream ID is a media link-local identifier for a stream. In other words, a particular stream ID must exist on both sides of a mediaDocumentation/userspace-api/media/v4l/dev-subdev.rst streams from one end of the link to the other, and subdevices have routing · line 554
Quoted 10 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +8
-1link, but another stream ID can be used for the same stream at the other side -of the subdevice. +of the subdevice. The same stream ID is used to refer to the stream on +both pads of the link on all ioctls operating on pads. + +A stream at a specific point in the media pipeline is identified by the +sub-devdev and a pad ID-stream ID pair. For subdevices that do not support +multiplexed streams the 'stream' field is always 0. + +Interaction between routes, formats and selectionss/selections/selection rectangles/
Quoted 7 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst streams from one end of the link to the other, and subdevices have routing · line 558
Quoted 4 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +4
-0+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The addition of routes to the V4L2 sub-device interface moves the +sub-device formats and selections from pads to pad ID-stream ID pairs.This reads as "pad" "ID-stream" "ID" "pairs". Even knowing how the API behaves, I had trouble understanding this.
Quoted 2 lines
Well, that's perhaps a bit exaggerated but I agree the readability could be improved.
I promise I wasn't exagerating, I had to re-read the sentence multiple times before getting what it meant :-)
Quoted 5 lines
Maybe:
(pad ID, stream ID) pair
? Not exactly great perhaps but should be easier to read.
How about
The addition of routes to the V4L2 sub-device interface moves the sub-device formats and selections from pads to (pad, stream) pairs.
? The text doesn't say "from pad IDs", so you could drop ID completely.
Quoted 30 lines
Quoted 10 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst streams from one end of the link to the other, and subdevices have routing · line 566
Quoted 8 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +8
-0+Besides the usual pad ID, also the stream ID needs to be provided for +setting formats and selections. + +Instead of the sub-device wide merging of streams from all source pads +towards all sink pads, data flows for each route are separate from each +other. Any number of routes from streams on sink pads towards streams on +source pads is allowed, to the extent supported by drivers. For every +stream on a sink pad, however, only a single route is allowed.This explains routes, not format and selection rectangles.
Perhaps a few more words on formats and selections would be appropriate indeed.
Quoted 16 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst streams from one end of the link to the other, and subdevices have routing · line 570
Quoted 6 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +3
-3-A stream at a specific point in the media pipeline is identified with the -subdev and a (pad, stream) pair. For subdevices that do not support -multiplexed streams the 'stream' is always 0. +Stream IDs, part of routes, are configured by using the +:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. This will +also revert the format and selection configuration to device defaults.I don't expect someone who doesn't have detailed knowledge of the implementation to understand what you mean here :-(
Quoted 6 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +0
-0Configuring streams ^^^^^^^^^^^^^^^^^^^@@ -565,8 +586,11 @@ Controller API <media_controller>` setting the routing table will reset all the stream configurations in a media entity.
Quoted 8 lines
Quoted 6 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst Controller API <media_controller>` · line 589
Quoted 3 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +1
-2-3) Configure streams. Each route endpoint must be configured -with :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>`. +3) Configure formats and selections. Each route is configured separatelyWe don't configure routes but streams, don't we ? Mentioning routes here seems confusing to me.
Well, routing, format and selection APIs all apply to streams, don't they?
Yes, but a route is a connection between two streams within a subdev, so formats and selection rectangles are not configured on routes but on streams.
Quoted 8 lines
I'll see if I can make this easier to grasp.
Quoted 3 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst Controller API <media_controller>` · line 590
Quoted 1 line
Documentation/userspace-api/media/v4l/dev-subdev.rst +1
-0+as documented plain subdevices in :ref:`<format-propagation>`. The streamDid you mean "as documented for plain subdevices" ?
Yes.
Quoted 8 lines
Quoted 4 lines
This is the part that bothers me. I don't think we can consider the configuration of routes to be independent in the general case. How the formats and selection rectangle of different streams interact with each other is what I'd like to see documented.
That is device specific in the end. Even without routes there are hardware specific limitations on what can be done, in this case it's just that other routes can be limiting as well.
That's right, but the V4L2 subdevice model nonetheless has a set of rules that govern how formats and selection rectangles interact with each other. Those rules are expected to apply to all devices and drivers.
In practice we have many drivers (in particular camera sensor drivers) that behave incorrectly. One could argue that this means the rules are not applicable, and we shouldn't try to carry this over to streams. For devices that require a custom userspace anyway (such as ISPs), this could be acceptable. For other devices, and in particular for camera sensor drivers, I strongly disagree. We need a stricter set of rules to make it possible to implement a generic userspace.
Quoted 10 lines
But as both are configured by the user, what takes priority, the pre-existing configuration or the new configuration set by the user? I think we indeed need rules for that.
Quoted 5 lines
As this is an area that has been entirely unexplored until now as far as I can tell, I would be fine if we wanted to merge streams support without support for selection rectangles to start with, and experiment with the selection API later, once we have good use cases that can be used for design and development.
Quoted 3 lines
Formats have dependencies, too, but perhaps a little less than if you add selections to the mix, too. In any case this should be documented in a different section.
You can pick the section in which you'd like to document this :-)
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +3
-0+ID is set to the same stream ID associated with either sink or source pads +of routes configured using the :ref:`VIDIOC_SUBDEV_S_ROUTING +<VIDIOC_SUBDEV_G_ROUTING>` ioctl. Multiplexed streams setup example ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Signature · 3 lines
Regards, Laurent Pinchart
Hi Laurent,
Quoted 87 lines · Laurent Pinchart
Hi Sakari,
Quoted 74 lines · Sakari Ailus
Quoted 65 lines · Laurent Pinchart
Quoted 51 lines · Sakari Ailus
Document how setting up routes interacts with formats and selections.
Signed-off-by Sakari Ailus <[email protected]>---
since v1:- Rewording changes based on Tomi's comments. In particular stricter separation between "stream" and "stream ID".
.../userspace-api/media/v4l/dev-subdev.rst | 50 ++++++++++++++----- 1 file changed, 37 insertions(+), 13 deletions(-)Documentation/userspace-api/media/v4l/dev-subdev.rst +9
-7@@ -406,6 +406,8 @@ pixel array is not rectangular but cross-shaped or round. The maximum size may also be smaller than the BOUNDS rectangle. +.. _format-propagation: + Order of configuration and format propagation ---------------------------------------------@@ -507,12 +509,12 @@ source pads. Streams, multiplexed media pads and internal routing ---------------------------------------------------- -Commonly V4L2 subdevices support only separate video streams, that is, only a -single stream can pass through a media link and a media pad. Thus each pad -contains a format configuration for that single stream. In some cases a subdev -can do stream processing and split a stream into two or compose two streams -into one, but the inputs and outputs for the subdev are still a single stream -per pad. +Simple V4L2 subdevices do not support multiple, unrelated video streams, +and only a single stream can pass through a media link and a media pad. +Thus each pad contains a format and selection configuration for that +single stream. A subdev can do stream processing and split a stream into +two or compose two streams into one, but the inputs and outputs for the +subdev are still a single stream per pad. Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is, multiple data streams are transmitted on the same bus, which is represented by a media@@ -539,14 +541,33 @@ streams from one end of the link to the other, and subdevices have routing tables which describe how the incoming streams from sink pads are routed to the source pads. -A stream ID (often just "stream") is a media link-local identifier for a stream. +A stream ID is a media link-local identifier for a stream. In other words, a particular stream ID must exist on both sides of a mediaDocumentation/userspace-api/media/v4l/dev-subdev.rst streams from one end of the link to the other, and subdevices have routing · line 554
Quoted 10 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +8
-1link, but another stream ID can be used for the same stream at the other side -of the subdevice. +of the subdevice. The same stream ID is used to refer to the stream on +both pads of the link on all ioctls operating on pads. + +A stream at a specific point in the media pipeline is identified by the +sub-devdev and a pad ID-stream ID pair. For subdevices that do not support +multiplexed streams the 'stream' field is always 0. + +Interaction between routes, formats and selectionss/selections/selection rectangles/
Quoted 7 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst streams from one end of the link to the other, and subdevices have routing · line 558
Quoted 4 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +4
-0+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The addition of routes to the V4L2 sub-device interface moves the +sub-device formats and selections from pads to pad ID-stream ID pairs.This reads as "pad" "ID-stream" "ID" "pairs". Even knowing how the API behaves, I had trouble understanding this.
Quoted 2 lines
Well, that's perhaps a bit exaggerated but I agree the readability could be improved.
I promise I wasn't exagerating, I had to re-read the sentence multiple times before getting what it meant :-)
Quoted 4 lines
Maybe:
(pad ID, stream ID) pair
Quoted 8 lines
Quoted 1 line
? Not exactly great perhaps but should be easier to read.
How about
The addition of routes to the V4L2 sub-device interface moves the sub-device formats and selections from pads to (pad, stream) pairs.
? The text doesn't say "from pad IDs", so you could drop ID completely.
That changes the meaning of it, for "stream" and "stream ID" are not the same things. I wonder if we could come up with another name for stream ID.
Quoted 38 lines
Quoted 38 lines
Quoted 11 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst streams from one end of the link to the other, and subdevices have routing · line 566
Quoted 9 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +8
-0+Besides the usual pad ID, also the stream ID needs to be provided for +setting formats and selections. + +Instead of the sub-device wide merging of streams from all source pads +towards all sink pads, data flows for each route are separate from each +other. Any number of routes from streams on sink pads towards streams on +source pads is allowed, to the extent supported by drivers. For every +stream on a sink pad, however, only a single route is allowed.This explains routes, not format and selection rectangles.
Perhaps a few more words on formats and selections would be appropriate indeed.
Quoted 22 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst streams from one end of the link to the other, and subdevices have routing · line 570
Quoted 6 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +3
-3-A stream at a specific point in the media pipeline is identified with the -subdev and a (pad, stream) pair. For subdevices that do not support -multiplexed streams the 'stream' is always 0. +Stream IDs, part of routes, are configured by using the +:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. This will +also revert the format and selection configuration to device defaults.I don't expect someone who doesn't have detailed knowledge of the implementation to understand what you mean here :-(
Documentation/userspace-api/media/v4l/dev-subdev.rst Controller API <media_controller>` · line 589
Quoted 9 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +1
-2Configuring streams ^^^^^^^^^^^^^^^^^^^@@ -565,8 +586,11 @@ Controller API <media_controller>` setting the routing table will reset all the stream configurations in a media entity. -3) Configure streams. Each route endpoint must be configured -with :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>`. +3) Configure formats and selections. Each route is configured separatelyWe don't configure routes but streams, don't we ? Mentioning routes here seems confusing to me.
Quoted 5 lines
Quoted 1 line
Well, routing, format and selection APIs all apply to streams, don't they?
Yes, but a route is a connection between two streams within a subdev, so formats and selection rectangles are not configured on routes but on streams.
It could be confusing for the reader to refer to streams only, as the formats (and selections) are propagated along the route. I'll see if I could include streams here.
Quoted 14 lines
Quoted 14 lines
I'll see if I can make this easier to grasp.
Quoted 3 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst Controller API <media_controller>` · line 590
Quoted 1 line
Documentation/userspace-api/media/v4l/dev-subdev.rst +1
-0+as documented plain subdevices in :ref:`<format-propagation>`. The streamDid you mean "as documented for plain subdevices" ?
Yes.
Quoted 4 lines
This is the part that bothers me. I don't think we can consider the configuration of routes to be independent in the general case. How the formats and selection rectangle of different streams interact with each other is what I'd like to see documented.
Quoted 8 lines
Quoted 3 lines
That is device specific in the end. Even without routes there are hardware specific limitations on what can be done, in this case it's just that other routes can be limiting as well.
That's right, but the V4L2 subdevice model nonetheless has a set of rules that govern how formats and selection rectangles interact with each other. Those rules are expected to apply to all devices and drivers.
Yes, but in this case the formats and selections do not depend only on other formats and selections earlier on in the chain, but those configured on other stream IDs on that pad as well. There's not much we can say about that while keeping the interface functional. The behaviour does need to be specified more precisely, such as telling which stream is a master (could be called e.g. cat) of another, in order to convey it is affected by configuration of another stream.
Quoted 32 lines
In practice we have many drivers (in particular camera sensor drivers) that behave incorrectly. One could argue that this means the rules are not applicable, and we shouldn't try to carry this over to streams. For devices that require a custom userspace anyway (such as ISPs), this could be acceptable. For other devices, and in particular for camera sensor drivers, I strongly disagree. We need a stricter set of rules to make it possible to implement a generic userspace.
Quoted 10 lines
But as both are configured by the user, what takes priority, the pre-existing configuration or the new configuration set by the user? I think we indeed need rules for that.
Quoted 5 lines
As this is an area that has been entirely unexplored until now as far as I can tell, I would be fine if we wanted to merge streams support without support for selection rectangles to start with, and experiment with the selection API later, once we have good use cases that can be used for design and development.
Quoted 3 lines
Formats have dependencies, too, but perhaps a little less than if you add selections to the mix, too. In any case this should be documented in a different section.
You can pick the section in which you'd like to document this :-)
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Documentation/userspace-api/media/v4l/dev-subdev.rst +3
-0+ID is set to the same stream ID associated with either sink or source pads +of routes configured using the :ref:`VIDIOC_SUBDEV_S_ROUTING +<VIDIOC_SUBDEV_G_ROUTING>` ioctl. Multiplexed streams setup example ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Signature · 3 lines
Kind regards, Sakari Ailus