s5p-fimc and noon010pc30 driver updates

Hi Mauro,

The following changes since commit f0a21151140da01c71de636f482f2eddec2840cc:

Merge tag 'v3.0' into staging/for_v3.1 (2011-07-22 13:33:14 -0300)

are available in the git repository at:

git://git.infradead.org/users/kmpark/linux-2.6-samsung fimc-for-mauro

Sylwester Nawrocki (28):
s5p-fimc: Add support for runtime PM in the mem-to-mem driver
s5p-fimc: Add media entity initialization
s5p-fimc: Remove registration of video nodes from probe()
s5p-fimc: Remove sclk_cam clock handling
s5p-fimc: Limit number of available inputs to one
s5p-fimc: Remove sensor management code from FIMC capture driver
s5p-fimc: Remove v4l2_device from video capture and m2m driver
s5p-fimc: Add the media device driver
s5p-fimc: Conversion to use struct v4l2_fh
s5p-fimc: Conversion to the control framework
s5p-fimc: Add media operations in the capture entity driver
s5p-fimc: Add PM helper function for streaming control
s5p-fimc: Correct color format enumeration
s5p-fimc: Convert to use media pipeline operations
s5p-fimc: Add subdev for the FIMC processing block
s5p-fimc: Add support for camera capture in JPEG format
s5p-fimc: Add v4l2_device notification support for single frame capture
s5p-fimc: Use consistent names for the buffer list functions
s5p-fimc: Add runtime PM support in the camera capture driver
s5p-fimc: Correct crop offset alignment on exynos4
s5p-fimc: Remove single-planar capability flags
noon010pc30: Do not ignore errors in initial controls setup
noon010pc30: Convert to the pad level ops
noon010pc30: Clean up the s_power callback
noon010pc30: Remove g_chip_ident operation handler
s5p-csis: Handle all available power supplies
s5p-csis: Rework of the system suspend/resume helpers
s5p-csis: Enable v4l subdev device node

From the last time you've submitted a similar set of patches:

Quoted 1 line

Why? The proper way to select an input is via S_INPUT. The driver

may also

Quoted 1 line

optionally allow changing it via the media device, but it should

not be

Quoted 1 line

a mandatory requirement, as the media device API is optional.

The problem I'm trying to solve here is sharing the sensors and mipi-csi receivers between multiple FIMC H/W instances. Previously the driver supported attaching a sensor to only one selected FIMC at compile time. You could, for instance, specify all sensors as the selected FIMC's platform data and then use S_INPUT to choose between them. The sensor could not be used together with any other FIMC. But this is desired due to different capabilities of the FIMC IP instances. And now, instead of hardcoding a sensor assigment to particular video node, the sensors are bound to the media device. The media device driver takes the list of sensors and attaches them one by one to subsequent FIMC instances when it is initializing. Each sensor has a link to each FIMC but only one of them is active by default. That said an user application can use selected camera by opening corresponding video node. Which camera is at which node can be queried with G_INPUT.

I could try to implement the previous S_INPUT behaviour, but IMHO this would lead to considerable and unnecessary driver code complication due to supporting overlapping APIs

From this current pull request:

From c6fb462c38be60a45d16a29a9e56c886ee0aa08c Mon Sep 17 00:00:00 2001

Hi Sylwester,

Em 27-07-2011 13:35, Sylwester Nawrocki escreveu:

Quoted 31 lines

Hi Mauro,

The following changes since commit f0a21151140da01c71de636f482f2eddec2840cc:

Merge tag 'v3.0' into staging/for_v3.1 (2011-07-22 13:33:14 -0300)

are available in the git repository at:

git://git.infradead.org/users/kmpark/linux-2.6-samsung fimc-for-mauro

Sylwester Nawrocki (28):
s5p-fimc: Add support for runtime PM in the mem-to-mem driver
s5p-fimc: Add media entity initialization
s5p-fimc: Remove registration of video nodes from probe()
s5p-fimc: Remove sclk_cam clock handling
s5p-fimc: Limit number of available inputs to one
s5p-fimc: Remove sensor management code from FIMC capture driver
s5p-fimc: Remove v4l2_device from video capture and m2m driver
s5p-fimc: Add the media device driver
s5p-fimc: Conversion to use struct v4l2_fh
s5p-fimc: Conversion to the control framework
s5p-fimc: Add media operations in the capture entity driver
s5p-fimc: Add PM helper function for streaming control
s5p-fimc: Correct color format enumeration
s5p-fimc: Convert to use media pipeline operations
s5p-fimc: Add subdev for the FIMC processing block
s5p-fimc: Add support for camera capture in JPEG format
s5p-fimc: Add v4l2_device notification support for single frame capture
s5p-fimc: Use consistent names for the buffer list functions
s5p-fimc: Add runtime PM support in the camera capture driver
s5p-fimc: Correct crop offset alignment on exynos4

Quoted 8 lines

s5p-fimc: Remove single-planar capability flags
noon010pc30: Do not ignore errors in initial controls setup
noon010pc30: Convert to the pad level ops
noon010pc30: Clean up the s_power callback
noon010pc30: Remove g_chip_ident operation handler
s5p-csis: Handle all available power supplies
s5p-csis: Rework of the system suspend/resume helpers
s5p-csis: Enable v4l subdev device node

From the last time you've submitted a similar set of patches:

Quoted 22 lines

Quoted 1 line

Why? The proper way to select an input is via S_INPUT. The driver

may also

Quoted 1 line

optionally allow changing it via the media device, but it should

not be

Quoted 1 line

a mandatory requirement, as the media device API is optional.

The problem I'm trying to solve here is sharing the sensors and mipi-csi receivers between multiple FIMC H/W instances. Previously the driver supported attaching a sensor to only one selected FIMC at compile time. You could, for instance, specify all sensors as the selected FIMC's platform data and then use S_INPUT to choose between them. The sensor could not be used together with any other FIMC. But this is desired due to different capabilities of the FIMC IP instances. And now, instead of hardcoding a sensor assigment to particular video node, the sensors are bound to the media device. The media device driver takes the list of sensors and attaches them one by one to subsequent FIMC instances when it is initializing. Each sensor has a link to each FIMC but only one of them is active by default. That said an user application can use selected camera by opening corresponding video node. Which camera is at which node can be queried with G_INPUT.

Quoted 3 lines

I could try to implement the previous S_INPUT behaviour, but IMHO this would lead to considerable and unnecessary driver code complication due to supporting overlapping APIs

From this current pull request:

From c6fb462c38be60a45d16a29a9e56c886ee0aa08c Mon Sep 17 00:00:00 2001
From: Sylwester Nawrocki <[email protected]>
Date: Fri, 10 Jun 2011 20:36:51 +0200
Subject: s5p-fimc: Conversion to the control framework
Cc: Linux Media Mailing List <[email protected]>

Make the driver inherit sensor controls when video node only API compatibility mode is enabled. The control framework allows to properly handle sensor controls through controls inheritance when pipeline is re-configured at media device level.

...

(quoted diff) +0 -3

-       .vidioc_queryctrl               = fimc_vidioc_queryctrl,
-       .vidioc_g_ctrl                  = fimc_vidioc_g_ctrl,
-       .vidioc_s_ctrl                  = fimc_cap_s_ctrl,
...

I'll need to take some time to review this patchset. So, it will likely miss the bus for 3.1.

While the code inside this patch looked ok, your comments scared me ;)

In summary: The V4L2 API is not a legacy API that needs a "compatibility mode". Removing controls like VIDIOC_S_INPUT, VIDIOC_*CTRL, etc in favor of the media controller API is wrong. This specific patch itself seems

ok, but it is easy to loose the big picture on a series of 28 patches with about 4000 lines changed.

The media controller API is meant to be used only by specific applications that might add some extra features to the driver. So, it is an optional API. In all cases where both API's can do the same thing, the proper way is to use the V4L2 API only, and not the media controller API.

So, my current plan is to merge the patches into an experimental tree, after reviewing the changeset, and test against a V4L2 application, in order to confirm that everything is ok.

I may need a couple weeks for doing that, as it will take some time for me to have an available window for hacking with it.

Hi Mauro,

Quoted 4 lines · Mauro Carvalho Chehab

Hi Sylwester,

Em 27-07-2011 13:35, Sylwester Nawrocki escreveu:

Quoted 1 line

Hi Mauro,

Quoted 3 lines

In summary: The V4L2 API is not a legacy API that needs a "compatibility mode". Removing controls like VIDIOC_S_INPUT, VIDIOC_*CTRL, etc in favor of the media controller API is wrong. This specific patch itself seems

Yes, it's the second time you say MC API is only optional;) I should had formulated the summary from the other point of view. I wrote this in context of two: V4L2 and MC API compatibility modes. Perhaps not too fortunate wording.

Quoted 2 lines

ok, but it is easy to loose the big picture on a series of 28 patches with about 4000 lines changed.

Yes, I agree. You really have to look carefully at the final result too.

When it comes to controls, as you might know, I didn't remove any functionality. Although the ioctl handlers are gotten rid of in the driver, they are handled in the control framework.

Quoted 4 lines

The media controller API is meant to be used only by specific applications that might add some extra features to the driver. So, it is an optional API. In all cases where both API's can do the same thing, the proper way is to use the V4L2 API only, and not the media controller API.

Yes I continually keep that in mind. But there are some corner cases when things are not so obvious, e.g. normally a video node inherits controls from a sensor subdev. So all the controls are available at the video node.

But when using the subdev API, the right thing to do at the video node is not to inherit sensor's controls. You could of course accumulate all controls at video node at all times, such as same (sensor subdev's) controls are available at /dev/video* and /dev/v4l-subdev*. But this is confusing to MC API aware application which could wrongly assume that there is more controls at the host device than there really is.

Thus it's a bit hard to imagine that we could do something like "optionally not to inherit controls" as the subdev/MC API is optional. :)

Also, the sensor subdev can be configured in the video node driver as well as through the subdev device node. Both APIs can do the same thing but in order to let the subdev API work as expected the video node driver must be forbidden to configure the subdev.

There is a conflict there that in order to use 'optional' API the 'main' API behaviour must be affected....

And I really cant use V4L2 API only as is because it's too limited.

Might be that's why I see more and more often migration to OpenMAX recently.

Quoted 3 lines

So, my current plan is to merge the patches into an experimental tree, after reviewing the changeset, and test against a V4L2 application, in order to confirm that everything is ok.

Sure, thanks. Basically I have been testing with same application before and after the patchset. Tomasz also tried his libv4l2 mplane patches with this reworked driver. So in general I do not expect any surprises,

but the testing can only help us:)

Quoted 3 lines

I may need a couple weeks for doing that, as it will take some time for me to have an available window for hacking with it.

At first I need to provide you with the board setup patches for smdkv310, in order to make m5mols camera work on this board. I depend on others to make this job and this is taking painfully long time. Maybe after the upcomning Linaro Connect I'll get things speed up.

Quoted 8 lines

Quoted 3 lines

In summary: The V4L2 API is not a legacy API that needs a "compatibility mode". Removing controls like VIDIOC_S_INPUT, VIDIOC_*CTRL, etc in favor of the media controller API is wrong. This specific patch itself seems

Yes, it's the second time you say MC API is only optional;) I should had formulated the summary from the other point of view. I wrote this in context of two: V4L2 and MC API compatibility modes. Perhaps not too fortunate wording.

A clear patch description helps for reviewers to understand what, why and how the patch is doing. Sometimes, I write a one-line patch, together with a 30+ lines of patch description. Especially on tricky patches, please be verbose at the descriptions.

Quoted 5 lines

Quoted 2 lines

ok, but it is easy to loose the big picture on a series of 28 patches with about 4000 lines changed.

Yes, I agree. You really have to look carefully at the final result too.

Quoted 3 lines

When it comes to controls, as you might know, I didn't remove any functionality. Although the ioctl handlers are gotten rid of in the driver, they are handled in the control framework.

Yes, I noticed, but, on a complex driver with several subdevs, it is not that simple to check where the controls are actually created, or if they require a MC API call to create them or not. Especially on patch series made by the

manufacturer, I generally don't spend much time to fully understand the driver logic, as I assume that the developers are doing the better to make the driver to work. My main concern is to be sure that the driver will be doing the right thing, in the light of the V4L2 API. The MC API made my work harder, as now, I need to check also if, for the device to work, it needs some MC API calls.

So, I have now 2 alternatives left:
a) to test with a device;
b) to fully understand the driver's logic.

Both are very time-consuming, but (a) is quicker, and safer, of course provided that I don't need to dig into several trees to get patches because not everything is not upstream yet.

This also means that I'll need the (complex) patches for devices with MC several weeks before the next merge window, to give me some time to open a window for testing.

Quoted 9 lines

Quoted 4 lines

The media controller API is meant to be used only by specific applications that might add some extra features to the driver. So, it is an optional API. In all cases where both API's can do the same thing, the proper way is to use the V4L2 API only, and not the media controller API.

Yes I continually keep that in mind. But there are some corner cases when things are not so obvious, e.g. normally a video node inherits controls from a sensor subdev. So all the controls are available at the video node.

Quoted 6 lines

But when using the subdev API, the right thing to do at the video node is not to inherit sensor's controls. You could of course accumulate all controls at video node at all times, such as same (sensor subdev's) controls are available at /dev/video* and /dev/v4l-subdev*. But this is confusing to MC API aware application which could wrongly assume that there is more controls at the host device than there really is.

Accumulating sub-dev controls at the video node is the right thing to do.

An MC-aware application will need to handle with that, but that doesn't sound to be hard. All such application would need to do is to first probe the subdev controls, and, when parsing the videodev controls, not register controls with duplicated ID's, or to mark them with some special attribute.

Quoted 3 lines

Thus it's a bit hard to imagine that we could do something like "optionally not to inherit controls" as the subdev/MC API is optional. :)

This was actually implemented. There are some cases at ivtv/cx18 driver where both the bridge and a subdev provides the same control (audio volume, for example). The idea is to allow the bridge driver to touch at the subdev control without exposing it to userspace, since the desire was that the bridge driver itself would expose such control, using a logic that combines changing the subdev and the bridge registers for volume.

Quoted 5 lines

Also, the sensor subdev can be configured in the video node driver as well as through the subdev device node. Both APIs can do the same thing but in order to let the subdev API work as expected the video node driver must be forbidden to configure the subdev.

Why? For the sensor, a V4L2 API call will look just like a bridge driver call. The subdev will need a mutex anyway, as two MC applications may be opening it simultaneously. I can't see why it should forbid changing the control from the bridge driver call.

Quoted 3 lines

There is a conflict there that in order to use 'optional' API the 'main' API behaviour must be affected....

It is optional from userspace perspective. A V4L2-only application should be able to work with all drivers. However, a MC-aware application will likely be specific for some hardware, as it will need to know some device-specific stuff.

Both kinds of applications are welcome, but dropping support for V4L2-only applications is the wrong thing to do.

Quoted 1 line

And I really cant use V4L2 API only as is because it's too limited.

Why?

Quoted 2 lines

Might be that's why I see more and more often migration to OpenMAX recently.

I don't think so. People may be adopting OpenMAX just because of some marketing strategy from the OpenMAX forum. We don't spend money to announce V4L2 ;)

I think that writing a pure OpenMAX driver is the wrong thing to do, as, at the long term, it will cost _a_lot_ for the vendors to maintain something that will never be merged upstream.

On the other hand, a V4L2/MC<==> OpenMAX abstraction layer/library at userspace makes sense, as it will open support for OpenMAX-aware userspace applications. Using an standard there would allow someone to write an application that would work on more than one operational system.

Yet, if I would be requested to write a multi-OS application, I would probably be opting to write an OS-specific driver for each OS, as it would allow the application to be optimized for that OS. The OS-specific layer is the small part of such application. Btw, successfully xawtv does that, supporting both V4L2 and a BSD-specific API. The size of the driver corresponds to about 2.5% of the total size of the application. So, writing two or tree drivers won't have any significant impact at the TCO of such application, especially because maintaining a generic multi-OS driver requires much more time for debugging/testing/maintaining than a per-OS driver. But, of course, this is a matter of developer's taste.

Quoted 7 lines

Quoted 3 lines

So, my current plan is to merge the patches into an experimental tree, after reviewing the changeset, and test against a V4L2 application, in order to confirm that everything is ok.

Sure, thanks. Basically I have been testing with same application before and after the patchset. Tomasz also tried his libv4l2 mplane patches with this reworked driver. So in general I do not expect any surprises,

Good to know.

Quoted 10 lines

but the testing can only help us:)

Quoted 3 lines

I may need a couple weeks for doing that, as it will take some time for me to have an available window for hacking with it.

At first I need to provide you with the board setup patches for smdkv310, in order to make m5mols camera work on this board. I depend on others to make this job and this is taking painfully long time. Maybe after the upcoming Linaro Connect I'll get things speed up.

Ok. I'll be waiting for you.

Thanks!
Mauro

Quoted 7 lines

Accumulating sub-dev controls at the video node is the right thing to do.

An MC-aware application will need to handle with that, but that doesn't sound to be hard. All such application would need to do is to first probe the subdev controls, and, when parsing the videodev controls, not register controls with duplicated ID's, or to mark them with some special attribute.

IMHO it's not a big issue in general. Still, both subdev and the host device may support same control id. And then even though the control ids are same on the subdev and the host they could mean physically different controls (since when registering a subdev at the host driver the host's controls take precedence and doubling subdev controls are skipped).

Also there might be some preference at user space, at which stage of the pipeline to apply some controls. This is where the subdev API helps, and plain video node API does not.

Quoted 10 lines

Quoted 3 lines

Thus it's a bit hard to imagine that we could do something like "optionally not to inherit controls" as the subdev/MC API is optional. :)

This was actually implemented. There are some cases at ivtv/cx18 driver where both the bridge and a subdev provides the same control (audio volume, for example). The idea is to allow the bridge driver to touch at the subdev control without exposing it to userspace, since the desire was that the bridge driver itself would expose such control, using a logic that combines changing the subdev and the bridge registers for volume.

This seem like hard coding a policy in the driver;) Then there is no way (it might not be worth the effort though) to play with volume level at both devices, e.g. to obtain optimal S/N ratio.

This is a hack...sorry, just joking ;-) Seriously, I think the situation with the userspace subdevs is a bit different. Because with one API we directly expose some functionality for applications, with other we code it in the kernel, to make the devices appear uniform at user space.

Quoted 9 lines

Quoted 4 lines

Also, the sensor subdev can be configured in the video node driver as well as through the subdev device node. Both APIs can do the same thing but in order to let the subdev API work as expected the video node driver must be forbidden to configure the subdev.

Why? For the sensor, a V4L2 API call will look just like a bridge driver call. The subdev will need a mutex anyway, as two MC applications may be opening it simultaneously. I can't see why it should forbid changing the control from the bridge driver call.

Please do not forget there might be more than one subdev to configure and that the bridge itself is also a subdev (which exposes a scaler interface, for instance). A situation pretty much like in Figure 4.4 [1] (after the scaler there is also a video node to configure, but we may assume that pixel resolution at the scaler pad 1 is same as at the video node). Assuming the format and crop configuration flow is from sensor to host scaler direction, if we have tried to configure _all_ subdevs when the last stage of the pipeline is configured (i.e. video node) the whole scaler and crop/composition configuration we have been destroyed at that time. And there is more to configure than VIDIOC_S_FMT can do.

Allowing the bridge driver to configure subdevs at all times would prevent the subdev/MC API to work.

Quoted 10 lines

Quoted 2 lines

There is a conflict there that in order to use 'optional' API the 'main' API behaviour must be affected....

It is optional from userspace perspective. A V4L2-only application should be able to work with all drivers. However, a MC-aware application will likely be specific for some hardware, as it will need to know some device-specific stuff.

Both kinds of applications are welcome, but dropping support for V4L2-only applications is the wrong thing to do.

Quoted 3 lines

Quoted 1 line

And I really cant use V4L2 API only as is because it's too limited.

Why?

For instance there is really yet no support for scaler and composition onto a target buffer in the Video Capture Interface (we also use sensors with built in scalers). It's difficult to efficiently manage capture/preview pipelines. It is impossible to discover the system topology.

Quoted 4 lines

Quoted 1 line

Might be that's why I see more and more often migration to OpenMAX recently.

I don't think so. People may be adopting OpenMAX just because of some marketing strategy from the OpenMAX forum. We don't spend money to announce V4L2 ;)

:)

Quoted 1 line

Quoted 3 lines

I think that writing a pure OpenMAX driver is the wrong thing to do, as, at the long term, it will cost _a_lot_ for the vendors to maintain something that will never be merged upstream.

In general it depends on priorities. For some chip vendors it might be more important to provide a solution in short time frame. Getting something in mainline isn't effortless and spending lot's of time on this for some parties is unacceptable.

Em 28-07-2011 19:57, Sylwester Nawrocki escreveu:

Quoted 10 lines · Mauro Carvalho Chehab

Quoted 8 lines

Accumulating sub-dev controls at the video node is the right thing to do.

An MC-aware application will need to handle with that, but that doesn't sound to be hard. All such application would need to do is to first probe the subdev controls, and, when parsing the videodev controls, not register controls with duplicated ID's, or to mark them with some special attribute.

Quoted 5 lines

IMHO it's not a big issue in general. Still, both subdev and the host device may support same control id. And then even though the control ids are same on the subdev and the host they could mean physically different controls (since when registering a subdev at the host driver the host's controls take precedence and doubling subdev controls are skipped).

True, but, except for specific usecases, the host control is enough.

Quoted 3 lines

Also there might be some preference at user space, at which stage of the pipeline to apply some controls. This is where the subdev API helps, and plain video node API does not.

Again, this is for specific usecases. On such cases, what is expected is that the more generic control will be exported via V4L2 API.

Quoted 12 lines

Quoted 11 lines

Quoted 3 lines

Thus it's a bit hard to imagine that we could do something like "optionally not to inherit controls" as the subdev/MC API is optional. :)

This was actually implemented. There are some cases at ivtv/cx18 driver where both the bridge and a subdev provides the same control (audio volume, for example). The idea is to allow the bridge driver to touch at the subdev control without exposing it to userspace, since the desire was that the bridge driver itself would expose such control, using a logic that combines changing the subdev and the bridge registers for volume.

Quoted 3 lines

This seem like hard coding a policy in the driver;) Then there is no way (it might not be worth the effort though) to play with volume level at both devices, e.g. to obtain optimal S/N ratio.

In general, playing with just one control is enough. Andy had a different opinion when this issue were discussed, and he thinks that playing with both is better. At the end, this is a developers decision, depending on how much information (and bug reports) he had.

Quoted 5 lines

This is a hack...sorry, just joking ;-) Seriously, I think the situation with the userspace subdevs is a bit different. Because with one API we directly expose some functionality for applications, with other we code it in the kernel, to make the devices appear uniform at user space.

Not sure if I understood you. V4L2 export drivers functionality to userspace in an uniform way. MC api is for special applications that might need to access some internal functions on embedded devices.

Of course, there are some cases where it doesn't make sense to export a subdev control via V4L2 API.

Quoted 10 lines

Quoted 9 lines

Quoted 4 lines

Also, the sensor subdev can be configured in the video node driver as well as through the subdev device node. Both APIs can do the same thing but in order to let the subdev API work as expected the video node driver must be forbidden to configure the subdev.

Why? For the sensor, a V4L2 API call will look just like a bridge driver call. The subdev will need a mutex anyway, as two MC applications may be opening it simultaneously. I can't see why it should forbid changing the control from the bridge driver call.

Quoted 10 lines

Please do not forget there might be more than one subdev to configure and that the bridge itself is also a subdev (which exposes a scaler interface, for instance). A situation pretty much like in Figure 4.4 [1] (after the scaler there is also a video node to configure, but we may assume that pixel resolution at the scaler pad 1 is same as at the video node). Assuming the format and crop configuration flow is from sensor to host scaler direction, if we have tried to configure _all_ subdevs when the last stage of the pipeline is configured (i.e. video node) the whole scaler and crop/composition configuration we have been destroyed at that time. And there is more to configure than VIDIOC_S_FMT can do.

Think from users perspective: all user wants is to see a video of a given resolution. S_FMT (and a few other VIDIOC_* calls) have everything that the user wants: the desired resolution, framerate and format.

Specialized applications indeed need more, in order to get the best images for certain types of usages. So, MC is there.

Such applications will probably need to know exactly what's the sensor, what are their bugs, how it is connected, what are the DSP blocks in the patch, how the DSP algorithms are implemented, etc, in order to obtain the the perfect image.

Even on embedded devices like smartphones and tablets, I predict that both types of applications will be developed and used: people may use a generic application like flash player, and an specialized application provided by the manufacturer. Users can even develop their own applications generic apps using V4L2 directly, at the devices that allow that.

As I said before: both application types are welcome. We just need to warrant that a pure V4L application will work reasonably well.

Quoted 2 lines

Allowing the bridge driver to configure subdevs at all times would prevent the subdev/MC API to work.

Well, then we need to think on an alternative for that. It seems an interesting theme for the media workshop at the Kernel Summit/2011.

Quoted 13 lines

Quoted 13 lines

Quoted 2 lines

There is a conflict there that in order to use 'optional' API the 'main' API behaviour must be affected....

It is optional from userspace perspective. A V4L2-only application should be able to work with all drivers. However, a MC-aware application will likely be specific for some hardware, as it will need to know some device-specific stuff.

Both kinds of applications are welcome, but dropping support for V4L2-only applications is the wrong thing to do.

Quoted 1 line

And I really cant use V4L2 API only as is because it's too limited.

Quoted 7 lines

Quoted 1 line

Why?

For instance there is really yet no support for scaler and composition onto a target buffer in the Video Capture Interface (we also use sensors with built in scalers). It's difficult to efficiently manage capture/preview pipelines. It is impossible to discover the system topology.

Scaler were always supported by V4L2: if the resolution specified by S_FMT is not what the sensor provides, then scale. All non-embedded drivers with sensor or bridge supports scale does that.

Composition is not properly supported yet. It could make sense to add it to V4L. How do you think MC API would help with composite?

Managing capture/preview pipelines will require some support at V4L2 level. This is a problem that needs to be addressed there anyway, as buffers for preview/capture need to be allocated. There's an RFC about that, but I don't think it covers the pipelines for it.

Discovering the system topology indeed is not part of V4L2 API and will never be. This is MC API business. There's no overlap with V4L2.

Quoted 9 lines

Quoted 7 lines

Quoted 2 lines

Might be that's why I see more and more often migration to OpenMAX recently.

I don't think so. People may be adopting OpenMAX just because of some marketing strategy from the OpenMAX forum. We don't spend money to announce V4L2 ;)

:)
:

Quoted 8 lines

Quoted 3 lines

I think that writing a pure OpenMAX driver is the wrong thing to do, as, at the long term, it will cost _a_lot_ for the vendors to maintain something that will never be merged upstream.

In general it depends on priorities. For some chip vendors it might be more important to provide a solution in short time frame. Getting something in mainline isn't effortless and spending lot's of time on this for some parties is unacceptable.

Adding something at a forum like OpenMAX is probably not an easy task ;) It generally takes a long time to change something on open forum specifications. Also, participating on those forums require lots of money, with membership and travel expenses.

Adding a new driver that follows the API's is not a long-term effort. The delay is basically one kernel cycle, e. g. about 3-4 months.

Most of the delays on merging drivers for embedded systems, however, take a way longer than that, unfortunately. From what I see, what is delaying driver submission is due to:
- the lack of how to work with the Linux community. Some developers take a
long time to get the 'modus operandi';
- the need of API changes. It is still generally faster to add a new API
addition at the Kernel than on an open forum;
- the discussions inside the various teams (generally from the same
company, or the company and their customers) about the better way to implement some feature.

All the above also happens when developing an OpenMAX driver: companies need to learn how to work with the OpenMax forums, API changes may be required, so they need to participate at the forums, the several internal teams and customers will be discussing the requirements.

I bet that there's also one additional step: to submit the driver to some company that will check the driver compliance with the official API. Such certification process is generally expensive and takes a long time.

At the end of the day, they'll also spend a lot of time to have the driver done, or they'll end by releasing a "not-quite-openmax" driver, and then needing to rework on that, due to customers complains, or to certification-compliance, loosing time and money.

Hi Mauro,

Quoted 17 lines · On Friday 29 July 2011 06:02:54 Mauro Carvalho Chehab wrote:

Em 28-07-2011 19:57, Sylwester Nawrocki escreveu:

Quoted 15 lines · Mauro Carvalho Chehab

Quoted 8 lines

Accumulating sub-dev controls at the video node is the right thing to do.

An MC-aware application will need to handle with that, but that doesn't sound to be hard. All such application would need to do is to first probe the subdev controls, and, when parsing the videodev controls, not register controls with duplicated ID's, or to mark them with some special attribute.

IMHO it's not a big issue in general. Still, both subdev and the host device may support same control id. And then even though the control ids are same on the subdev and the host they could mean physically different controls (since when registering a subdev at the host driver the host's controls take precedence and doubling subdev controls are skipped).

Quoted 1 line

True, but, except for specific usecases, the host control is enough.

Not for embedded devices. In many case the control won't even be implemented by the host. If your system has two sensors connected to the same host, they will both expose an exposure time control. Which one do you configure through the video node ? The two sensors will likely have different bounds for the same control, how do you report that ?

Those controls are also quite useless for generic V4L2 applications, which will want auto-exposure anyway. This needs to be implemented in userspace in libv4l.

Quoted 19 lines

Quoted 3 lines

Also there might be some preference at user space, at which stage of the pipeline to apply some controls. This is where the subdev API helps, and plain video node API does not.

Again, this is for specific usecases. On such cases, what is expected is that the more generic control will be exported via V4L2 API.

Quoted 12 lines

Quoted 11 lines

Quoted 3 lines

Thus it's a bit hard to imagine that we could do something like "optionally not to inherit controls" as the subdev/MC API is optional. :)

This was actually implemented. There are some cases at ivtv/cx18 driver where both the bridge and a subdev provides the same control (audio volume, for example). The idea is to allow the bridge driver to touch at the subdev control without exposing it to userspace, since the desire was that the bridge driver itself would expose such control, using a logic that combines changing the subdev and the bridge registers for volume.

Quoted 8 lines

Quoted 3 lines

This seem like hard coding a policy in the driver;) Then there is no way (it might not be worth the effort though) to play with volume level at both devices, e.g. to obtain optimal S/N ratio.

In general, playing with just one control is enough. Andy had a different opinion when this issue were discussed, and he thinks that playing with both is better. At the end, this is a developers decision, depending on how much information (and bug reports) he had.

ivtv/cx18 is a completely different use case, where hardware configurations are known, and experiments possible to find out which control(s) to use and how. In this case you can't know in advance what the sensor and host drivers will be used for.

Even if you did, fine image quality tuning requires accessing pretty much all controls individually anyway.

Quoted 52 lines

Quoted 5 lines

This is a hack...sorry, just joking ;-) Seriously, I think the situation with the userspace subdevs is a bit different. Because with one API we directly expose some functionality for applications, with other we code it in the kernel, to make the devices appear uniform at user space.

Not sure if I understood you. V4L2 export drivers functionality to userspace in an uniform way. MC api is for special applications that might need to access some internal functions on embedded devices.

Of course, there are some cases where it doesn't make sense to export a subdev control via V4L2 API.

Quoted 10 lines

Quoted 9 lines

Quoted 4 lines

Also, the sensor subdev can be configured in the video node driver as well as through the subdev device node. Both APIs can do the same thing but in order to let the subdev API work as expected the video node driver must be forbidden to configure the subdev.

Why? For the sensor, a V4L2 API call will look just like a bridge driver call. The subdev will need a mutex anyway, as two MC applications may be opening it simultaneously. I can't see why it should forbid changing the control from the bridge driver call.

Quoted 10 lines

Please do not forget there might be more than one subdev to configure and that the bridge itself is also a subdev (which exposes a scaler interface, for instance). A situation pretty much like in Figure 4.4 [1] (after the scaler there is also a video node to configure, but we may assume that pixel resolution at the scaler pad 1 is same as at the video node). Assuming the format and crop configuration flow is from sensor to host scaler direction, if we have tried to configure _all_ subdevs when the last stage of the pipeline is configured (i.e. video node) the whole scaler and crop/composition configuration we have been destroyed at that time. And there is more to configure than VIDIOC_S_FMT can do.

Think from users perspective: all user wants is to see a video of a given resolution. S_FMT (and a few other VIDIOC_* calls) have everything that the user wants: the desired resolution, framerate and format.

Specialized applications indeed need more, in order to get the best images for certain types of usages. So, MC is there.

Such applications will probably need to know exactly what's the sensor, what are their bugs, how it is connected, what are the DSP blocks in the patch, how the DSP algorithms are implemented, etc, in order to obtain the the perfect image.

Even on embedded devices like smartphones and tablets, I predict that both types of applications will be developed and used: people may use a generic application like flash player, and an specialized application provided by the manufacturer. Users can even develop their own applications generic apps using V4L2 directly, at the devices that allow that.

Quoted 2 lines

As I said before: both application types are welcome. We just need to warrant that a pure V4L application will work reasonably well.

That's why we have libv4l. The driver simply doesn't receive enough information to configure the hardware correctly from the VIDIOC_* calls. And as mentioned above, 3A algorithms, required by "simple" V4L2 applications, need to be implemented in userspace anyway.

Quoted 34 lines

Quoted 2 lines

Allowing the bridge driver to configure subdevs at all times would prevent the subdev/MC API to work.

Well, then we need to think on an alternative for that. It seems an interesting theme for the media workshop at the Kernel Summit/2011.

Quoted 13 lines

Quoted 13 lines

Quoted 2 lines

There is a conflict there that in order to use 'optional' API the 'main' API behaviour must be affected....

It is optional from userspace perspective. A V4L2-only application should be able to work with all drivers. However, a MC-aware application will likely be specific for some hardware, as it will need to know some device-specific stuff.

Both kinds of applications are welcome, but dropping support for V4L2-only applications is the wrong thing to do.

Quoted 1 line

And I really cant use V4L2 API only as is because it's too limited.

Quoted 7 lines

Quoted 1 line

Why?

For instance there is really yet no support for scaler and composition onto a target buffer in the Video Capture Interface (we also use sensors with built in scalers). It's difficult to efficiently manage capture/preview pipelines. It is impossible to discover the system topology.

Scaler were always supported by V4L2: if the resolution specified by S_FMT is not what the sensor provides, then scale. All non-embedded drivers with sensor or bridge supports scale does that.

Composition is not properly supported yet. It could make sense to add it to V4L. How do you think MC API would help with composite?

Quoted 4 lines

Managing capture/preview pipelines will require some support at V4L2 level. This is a problem that needs to be addressed there anyway, as buffers for preview/capture need to be allocated. There's an RFC about that, but I don't think it covers the pipelines for it.

Managing pipelines is a policy decision, and needs to be implemented in userspace. Once again, the solution here is libv4l.

Quoted 21 lines

Discovering the system topology indeed is not part of V4L2 API and will never be. This is MC API business. There's no overlap with V4L2.

Quoted 9 lines

Quoted 7 lines

Quoted 2 lines

Might be that's why I see more and more often migration to OpenMAX recently.

I don't think so. People may be adopting OpenMAX just because of some marketing strategy from the OpenMAX forum. We don't spend money to announce V4L2 ;)

:)
:

Quoted 8 lines

Quoted 3 lines

I think that writing a pure OpenMAX driver is the wrong thing to do, as, at the long term, it will cost _a_lot_ for the vendors to maintain something that will never be merged upstream.

In general it depends on priorities. For some chip vendors it might be more important to provide a solution in short time frame. Getting something in mainline isn't effortless and spending lot's of time on this for some parties is unacceptable.

Quoted 4 lines

Adding something at a forum like OpenMAX is probably not an easy task ;) It generally takes a long time to change something on open forum specifications. Also, participating on those forums require lots of money, with membership and travel expenses.

Correct me if I'm wrong (I'm not an OpenMAX specialist), but I think it doesn't. OpenMAX isn't really open, and vendors implement proprietary extensions as needed. My understanding is that tt's easier for them to use OpenMAX for that, because they can more or less claim OpenMAX support even with the proprietary extensions, something they couldn't do with V4L2.

If we want to compete there (and I certainly do), we need to provide vendors with clear, easy and well documented APIs that fulfill their needs out of the box. That's of course an evermoving target.

Quoted 15 lines

Adding a new driver that follows the API's is not a long-term effort. The delay is basically one kernel cycle, e. g. about 3-4 months.

Most of the delays on merging drivers for embedded systems, however, take a way longer than that, unfortunately. From what I see, what is delaying driver submission is due to:
- the lack of how to work with the Linux community. Some developers take a
long time to get the 'modus operandi';
- the need of API changes. It is still generally faster to add a new API
addition at the Kernel than on an open forum;
- the discussions inside the various teams (generally from the same
company, or the company and their customers) about the better way to implement some feature.

Quoted 8 lines

All the above also happens when developing an OpenMAX driver: companies need to learn how to work with the OpenMax forums, API changes may be required, so they need to participate at the forums, the several internal teams and customers will be discussing the requirements.

I bet that there's also one additional step: to submit the driver to some company that will check the driver compliance with the official API. Such certification process is generally expensive and takes a long time.

I could be wrong, but I very much doubt this ever happens.

Quoted 4 lines

At the end of the day, they'll also spend a lot of time to have the driver done, or they'll end by releasing a "not-quite-openmax" driver, and then needing to rework on that, due to customers complains, or to certification-compliance, loosing time and money.

Em 29-07-2011 05:36, Laurent Pinchart escreveu:

Quoted 20 lines

Hi Mauro,

Quoted 17 lines · On Friday 29 July 2011 06:02:54 Mauro Carvalho Chehab wrote:

Em 28-07-2011 19:57, Sylwester Nawrocki escreveu:

Quoted 15 lines · Mauro Carvalho Chehab

Quoted 8 lines

Accumulating sub-dev controls at the video node is the right thing to do.

An MC-aware application will need to handle with that, but that doesn't sound to be hard. All such application would need to do is to first probe the subdev controls, and, when parsing the videodev controls, not register controls with duplicated ID's, or to mark them with some special attribute.

IMHO it's not a big issue in general. Still, both subdev and the host device may support same control id. And then even though the control ids are same on the subdev and the host they could mean physically different controls (since when registering a subdev at the host driver the host's controls take precedence and doubling subdev controls are skipped).

Quoted 7 lines

Quoted 1 line

True, but, except for specific usecases, the host control is enough.

Not for embedded devices. In many case the control won't even be implemented by the host. If your system has two sensors connected to the same host, they will both expose an exposure time control. Which one do you configure through the video node ? The two sensors will likely have different bounds for the same control, how do you report that ?

If the device has two sensors that are mutually exclusive, they should be mapped as two different inputs. The showed control should be the one used by the currently active input.

If the sensors aren't mutually exclusive, then two different video nodes will be shown in userspace.

Quoted 3 lines

Those controls are also quite useless for generic V4L2 applications, which will want auto-exposure anyway. This needs to be implemented in userspace in libv4l.

Several webcams export exposure controls. Why shouldn't those controls be exposed to userspace anymore?

Ok, if the hardware won't support 3A algorithm, libv4l will implement it, eventually using an extra hardware-aware code to get the best performance for that specific device, but this doesn't mean that the user should always use it.

Btw, the 3A algorithm is one of the things I don't like on my cell phone: while it works most of the time, sometimes I want to disable it and manually adjust, as it produces dark images, when there's a very bright light somewhere on the image background. Manually adjusting the exposure time and aperture is something relevant for some users.

Quoted 28 lines

Quoted 19 lines

Quoted 3 lines

Also there might be some preference at user space, at which stage of the pipeline to apply some controls. This is where the subdev API helps, and plain video node API does not.

Again, this is for specific usecases. On such cases, what is expected is that the more generic control will be exported via V4L2 API.

Quoted 12 lines

Quoted 11 lines

Quoted 3 lines

Thus it's a bit hard to imagine that we could do something like "optionally not to inherit controls" as the subdev/MC API is optional. :)

This was actually implemented. There are some cases at ivtv/cx18 driver where both the bridge and a subdev provides the same control (audio volume, for example). The idea is to allow the bridge driver to touch at the subdev control without exposing it to userspace, since the desire was that the bridge driver itself would expose such control, using a logic that combines changing the subdev and the bridge registers for volume.

Quoted 8 lines

Quoted 3 lines

This seem like hard coding a policy in the driver;) Then there is no way (it might not be worth the effort though) to play with volume level at both devices, e.g. to obtain optimal S/N ratio.

In general, playing with just one control is enough. Andy had a different opinion when this issue were discussed, and he thinks that playing with both is better. At the end, this is a developers decision, depending on how much information (and bug reports) he had.

Quoted 4 lines

ivtv/cx18 is a completely different use case, where hardware configurations are known, and experiments possible to find out which control(s) to use and how. In this case you can't know in advance what the sensor and host drivers will be used for.

Why not? I never saw an embedded hardware that allows physically changing the sensor.

Quoted 2 lines

Even if you did, fine image quality tuning requires accessing pretty much all controls individually anyway.

The same is also true for non-embedded hardware. The only situation where V4L2 API is not enough is when there are two controls of the same type active. For example, 2 active volume controls, one at the audio demod, and another at the bridge. There may have some cases where you can do the same thing at the sensor or at a DSP block. This is where MC API gives an improvement, by allowing changing both, instead of just one of the controls.

Quoted 55 lines

Quoted 54 lines

Quoted 5 lines

This is a hack...sorry, just joking ;-) Seriously, I think the situation with the userspace subdevs is a bit different. Because with one API we directly expose some functionality for applications, with other we code it in the kernel, to make the devices appear uniform at user space.

Not sure if I understood you. V4L2 export drivers functionality to userspace in an uniform way. MC api is for special applications that might need to access some internal functions on embedded devices.

Of course, there are some cases where it doesn't make sense to export a subdev control via V4L2 API.

Quoted 10 lines

Quoted 9 lines

Quoted 4 lines

Also, the sensor subdev can be configured in the video node driver as well as through the subdev device node. Both APIs can do the same thing but in order to let the subdev API work as expected the video node driver must be forbidden to configure the subdev.

Why? For the sensor, a V4L2 API call will look just like a bridge driver call. The subdev will need a mutex anyway, as two MC applications may be opening it simultaneously. I can't see why it should forbid changing the control from the bridge driver call.

Quoted 10 lines

Please do not forget there might be more than one subdev to configure and that the bridge itself is also a subdev (which exposes a scaler interface, for instance). A situation pretty much like in Figure 4.4 [1] (after the scaler there is also a video node to configure, but we may assume that pixel resolution at the scaler pad 1 is same as at the video node). Assuming the format and crop configuration flow is from sensor to host scaler direction, if we have tried to configure _all_ subdevs when the last stage of the pipeline is configured (i.e. video node) the whole scaler and crop/composition configuration we have been destroyed at that time. And there is more to configure than VIDIOC_S_FMT can do.

Think from users perspective: all user wants is to see a video of a given resolution. S_FMT (and a few other VIDIOC_* calls) have everything that the user wants: the desired resolution, framerate and format.

Specialized applications indeed need more, in order to get the best images for certain types of usages. So, MC is there.

Such applications will probably need to know exactly what's the sensor, what are their bugs, how it is connected, what are the DSP blocks in the patch, how the DSP algorithms are implemented, etc, in order to obtain the the perfect image.

Even on embedded devices like smartphones and tablets, I predict that both types of applications will be developed and used: people may use a generic application like flash player, and an specialized application provided by the manufacturer. Users can even develop their own applications generic apps using V4L2 directly, at the devices that allow that.

As I said before: both application types are welcome. We just need to warrant that a pure V4L application will work reasonably well.

Quoted 4 lines

That's why we have libv4l. The driver simply doesn't receive enough information to configure the hardware correctly from the VIDIOC_* calls. And as mentioned above, 3A algorithms, required by "simple" V4L2 applications, need to be implemented in userspace anyway.

It is OK to improve users experience via libv4l. What I'm saying is that it is NOT OK to remove V4L2 API support from the driver, forcing users to use some hardware plugin at libv4l.

Quoted 39 lines

Quoted 38 lines

Quoted 2 lines

Allowing the bridge driver to configure subdevs at all times would prevent the subdev/MC API to work.

Well, then we need to think on an alternative for that. It seems an interesting theme for the media workshop at the Kernel Summit/2011.

Quoted 13 lines

Quoted 13 lines

Quoted 2 lines

There is a conflict there that in order to use 'optional' API the 'main' API behaviour must be affected....

It is optional from userspace perspective. A V4L2-only application should be able to work with all drivers. However, a MC-aware application will likely be specific for some hardware, as it will need to know some device-specific stuff.

Both kinds of applications are welcome, but dropping support for V4L2-only applications is the wrong thing to do.

Quoted 1 line

And I really cant use V4L2 API only as is because it's too limited.

Quoted 7 lines

Quoted 1 line

Why?

For instance there is really yet no support for scaler and composition onto a target buffer in the Video Capture Interface (we also use sensors with built in scalers). It's difficult to efficiently manage capture/preview pipelines. It is impossible to discover the system topology.

Scaler were always supported by V4L2: if the resolution specified by S_FMT is not what the sensor provides, then scale. All non-embedded drivers with sensor or bridge supports scale does that.

Composition is not properly supported yet. It could make sense to add it to V4L. How do you think MC API would help with composite?

Managing capture/preview pipelines will require some support at V4L2 level. This is a problem that needs to be addressed there anyway, as buffers for preview/capture need to be allocated. There's an RFC about that, but I don't think it covers the pipelines for it.

Quoted 2 lines

Managing pipelines is a policy decision, and needs to be implemented in userspace. Once again, the solution here is libv4l.

If V4L2 API is not enough, implementing it on libv4l won't solve, as userspace apps will use V4L2 API for requresting it.

Hi Mauro,

Quoted 29 lines · Mauro Carvalho Chehab

Em 29-07-2011 05:36, Laurent Pinchart escreveu:

Quoted 27 lines

Hi Mauro,

Quoted 11 lines · On Friday 29 July 2011 06:02:54 Mauro Carvalho Chehab wrote:

Em 28-07-2011 19:57, Sylwester Nawrocki escreveu:

Quoted 10 lines · Mauro Carvalho Chehab

Quoted 8 lines

Accumulating sub-dev controls at the video node is the right thing to do.

An MC-aware application will need to handle with that, but that doesn't sound to be hard. All such application would need to do is to first probe the subdev controls, and, when parsing the videodev controls, not register controls with duplicated ID's, or to mark them with some special attribute.

Quoted 7 lines

Quoted 5 lines

IMHO it's not a big issue in general. Still, both subdev and the host device may support same control id. And then even though the control ids are same on the subdev and the host they could mean physically different controls (since when registering a subdev at the host driver the host's controls take precedence and doubling subdev controls are skipped).

True, but, except for specific usecases, the host control is enough.

Not for embedded devices. In many case the control won't even be implemented by the host. If your system has two sensors connected to the same host, they will both expose an exposure time control. Which one do you configure through the video node ? The two sensors will likely have different bounds for the same control, how do you report that ?

Quoted 3 lines

If the device has two sensors that are mutually exclusive, they should be mapped as two different inputs. The showed control should be the one used by the currently active input.

Video nodes should represent a dma engine rather than an image source.

You
could use the same hardware to process data from memory to memory while streaming video from a sensor to memory at the same time. This is a quite typical use in embedded systems.

Usually boards where two sensors are connected to an isp the dependencies are not as straightforward as the sensors being fully independent or require exclusive access. Typically some of the hardware blocks are shared between the two and can be used by only one sensor at a time, so instead you may not get full functionality from both at the same time. And you need to be able to choose which one uses that hardware block. This is exactly what Media controller interface models perfectly.

See FIMC Media controller graph AND Sylwester's explanation on it; a few links are actually missing from the grapg.

<URL:http://www.spinics.net/lists/linux-media/msg35504.html> <URL:http://wstaw.org/m/2011/05/26/fimc_graph__.png>

Cc Hans.

Quoted 3 lines

If the sensors aren't mutually exclusive, then two different video nodes will be shown in userspace.

Quoted 6 lines

Quoted 3 lines

Those controls are also quite useless for generic V4L2 applications, which will want auto-exposure anyway. This needs to be implemented in userspace in libv4l.

Several webcams export exposure controls. Why shouldn't those controls be exposed to userspace anymore?

This is not a webcam,

it is a software controlled high end digital camera on
a mobile device. The difference is that the functionality offered by the hardware is at much lower level compared to a webcam; the result is that more detailed control ia required but also much flexibility and performance is gained.

Quoted 4 lines

Ok, if the hardware won't support 3A algorithm, libv4l will implement it, eventually using an extra hardware-aware code to get the best performance for that specific device, but this doesn't mean that the user should always use it.

Why not? What would be the alternative?

Quoted 5 lines

Btw, the 3A algorithm is one of the things I don't like on my cell phone: while it works most of the time, sometimes I want to disable it and manually adjust, as it produces dark images, when there's a very bright light somewhere on the image background. Manually adjusting the exposure time and aperture is something relevant for some users.

You do want the 3A algorithms even if you use manual white balance. What the automatic white balance algorithm produces is (among other things) gamma tables, rgb-to-rgb conversion matrices and lens shading correction tables. I doubt any end user, even if it was you, would like to fiddle with such large tables directly when capturing photos.

The size of this configuration could
easily be around 10 kB. A higher level control is required; colour temperature, for instance. And its implementation involves the same automatic white balance algorithm.

You must know your hardware very, very well to use the aforementioned low level controls and in such a case you have no reason not to use the MC interface to configure the device either. Configuring the device image pipe using MC is actually a number of magnitudes less complex, and I say it's been quite an achievement that we have such an interface which makes it so effortless to do.

Quoted 39 lines

Quoted 28 lines

Quoted 19 lines

Quoted 3 lines

Also there might be some preference at user space, at which stage of the pipeline to apply some controls. This is where the subdev API helps, and plain video node API does not.

Again, this is for specific usecases. On such cases, what is expected is that the more generic control will be exported via V4L2 API.

Quoted 12 lines

Quoted 11 lines

Quoted 3 lines

Thus it's a bit hard to imagine that we could do something like "optionally not to inherit controls" as the subdev/MC API is optional. :)

This was actually implemented. There are some cases at ivtv/cx18 driver where both the bridge and a subdev provides the same control (audio volume, for example). The idea is to allow the bridge driver to touch at the subdev control without exposing it to userspace, since the desire was that the bridge driver itself would expose such control, using a logic that combines changing the subdev and the bridge registers for volume.

Quoted 8 lines

Quoted 3 lines

This seem like hard coding a policy in the driver;) Then there is no way (it might not be worth the effort though) to play with volume level at both devices, e.g. to obtain optimal S/N ratio.

In general, playing with just one control is enough. Andy had a different opinion when this issue were discussed, and he thinks that playing with both is better. At the end, this is a developers decision, depending on how much information (and bug reports) he had.

Quoted 4 lines

ivtv/cx18 is a completely different use case, where hardware configurations are known, and experiments possible to find out which control(s) to use and how. In this case you can't know in advance what the sensor and host drivers will be used for.

Why not? I never saw an embedded hardware that allows physically changing the sensor.

Quoted 2 lines

Even if you did, fine image quality tuning requires accessing pretty much all controls individually anyway.

Quoted 7 lines

The same is also true for non-embedded hardware. The only situation where V4L2 API is not enough is when there are two controls of the same type active. For example, 2 active volume controls, one at the audio demod, and another at the bridge. There may have some cases where you can do the same thing at the sensor or at a DSP block. This is where MC API gives an improvement, by allowing changing both, instead of just one of the controls.

This may be true on non-embedded hardware. It's important to know which hardware component implements a particular control; for example digital gain typically would take longer to have an effect if it is set on sensor rather than on the ISP. Also, is you set digital gain, you want to set it on the same device where your analog gain is --- the sensor --- to avoid badly exposed imagws.

When it comes to scaling, the scaling quality, power consumption and performance may well be very different depending on where it is done. There typically are data rate limitations at different parts of the pipeline. The plain V4L2 has never been meant for this nor provides any support when doing something like above, and these are just few examples.

Quoted 55 lines

Quoted 55 lines

Quoted 54 lines

Quoted 5 lines

This is a hack...sorry, just joking ;-) Seriously, I think the situation with the userspace subdevs is a bit different. Because with one API we directly expose some functionality for applications, with other we code it in the kernel, to make the devices appear uniform at user space.

Not sure if I understood you. V4L2 export drivers functionality to userspace in an uniform way. MC api is for special applications that might need to access some internal functions on embedded devices.

Of course, there are some cases where it doesn't make sense to export a subdev control via V4L2 API.

Quoted 10 lines

Quoted 9 lines

Quoted 4 lines

Also, the sensor subdev can be configured in the video node driver as well as through the subdev device node. Both APIs can do the same thing but in order to let the subdev API work as expected the video node driver must be forbidden to configure the subdev.

Why? For the sensor, a V4L2 API call will look just like a bridge driver call. The subdev will need a mutex anyway, as two MC applications may be opening it simultaneously. I can't see why it should forbid changing the control from the bridge driver call.

Quoted 10 lines

Please do not forget there might be more than one subdev to configure and that the bridge itself is also a subdev (which exposes a scaler interface, for instance). A situation pretty much like in Figure 4.4 [1] (after the scaler there is also a video node to configure, but we may assume that pixel resolution at the scaler pad 1 is same as at the video node). Assuming the format and crop configuration flow is from sensor to host scaler direction, if we have tried to configure _all_ subdevs when the last stage of the pipeline is configured (i.e. video node) the whole scaler and crop/composition configuration we have been destroyed at that time. And there is more to configure than VIDIOC_S_FMT can do.

Think from users perspective: all user wants is to see a video of a given resolution. S_FMT (and a few other VIDIOC_* calls) have everything that the user wants: the desired resolution, framerate and format.

Specialized applications indeed need more, in order to get the best images for certain types of usages. So, MC is there.

Such applications will probably need to know exactly what's the sensor, what are their bugs, how it is connected, what are the DSP blocks in the patch, how the DSP algorithms are implemented, etc, in order to obtain the the perfect image.

Even on embedded devices like smartphones and tablets, I predict that both types of applications will be developed and used: people may use a generic application like flash player, and an specialized application provided by the manufacturer. Users can even develop their own applications generic apps using V4L2 directly, at the devices that allow that.

As I said before: both application types are welcome. We just need to warrant that a pure V4L application will work reasonably well.

Quoted 8 lines

Quoted 4 lines

That's why we have libv4l. The driver simply doesn't receive enough information to configure the hardware correctly from the VIDIOC_* calls. And as mentioned above, 3A algorithms, required by "simple" V4L2 applications, need to be implemented in userspace anyway.

It is OK to improve users experience via libv4l. What I'm saying is that it is NOT OK to remove V4L2 API support from the driver, forcing users to use some hardware plugin at libv4l.

Either do you know your hardware or do not know it. General purpose applications can rely on functionality provided by libv4l, but if you do not use it, then you need to configure the underlying device. Which is something where the Media controller and v4l2_subdev interfaces are tremendously useful.

Quoted 42 lines

Quoted 41 lines

Quoted 31 lines

Quoted 2 lines

Allowing the bridge driver to configure subdevs at all times would prevent the subdev/MC API to work.

Well, then we need to think on an alternative for that. It seems an interesting theme for the media workshop at the Kernel Summit/2011.

Quoted 13 lines

Quoted 13 lines

Quoted 2 lines

There is a conflict there that in order to use 'optional' API the 'main' API behaviour must be affected....

It is optional from userspace perspective. A V4L2-only application should be able to work with all drivers. However, a MC-aware application will likely be specific for some hardware, as it will need to know some device-specific stuff.

Both kinds of applications are welcome, but dropping support for V4L2-only applications is the wrong thing to do.

Quoted 1 line

And I really cant use V4L2 API only as is because it's too limited.

Quoted 7 lines

Quoted 1 line

Why?

For instance there is really yet no support for scaler and composition onto a target buffer in the Video Capture Interface (we also use sensors with built in scalers). It's difficult to efficiently manage capture/preview pipelines. It is impossible to discover the system topology.

Scaler were always supported by V4L2: if the resolution specified by S_FMT is not what the sensor provides, then scale. All non-embedded drivers with sensor or bridge supports scale does that.

Quoted 7 lines

Composition is not properly supported yet. It could make sense to add it to V4L. How do you think MC API would help with composite?

Managing capture/preview pipelines will require some support at V4L2 level. This is a problem that needs to be addressed there anyway, as buffers for preview/capture need to be allocated. There's an RFC about that, but I don't think it covers the pipelines for it.

Managing pipelines is a policy decision, and needs to be implemented in userspace. Once again, the solution here is libv4l.

Quoted 2 lines

If V4L2 API is not enough, implementing it on libv4l won't solve, as userspace apps will use V4L2 API for requresting it.

There are two kind of applications: specialised and generic. The generic ones may rely on restrictive policies put in place by a libv4l plugin whereas the specialised applications need to access the device's features directly to get the most out of it.

Em 09-08-2011 20:18, Sakari Ailus escreveu:

Quoted 32 lines

Hi Mauro,

Quoted 29 lines · Mauro Carvalho Chehab

Em 29-07-2011 05:36, Laurent Pinchart escreveu:

Quoted 27 lines

Hi Mauro,

Quoted 11 lines · On Friday 29 July 2011 06:02:54 Mauro Carvalho Chehab wrote:

Em 28-07-2011 19:57, Sylwester Nawrocki escreveu:

Quoted 10 lines · Mauro Carvalho Chehab

Quoted 8 lines

Accumulating sub-dev controls at the video node is the right thing to do.

An MC-aware application will need to handle with that, but that doesn't sound to be hard. All such application would need to do is to first probe the subdev controls, and, when parsing the videodev controls, not register controls with duplicated ID's, or to mark them with some special attribute.

Quoted 7 lines

Quoted 5 lines

IMHO it's not a big issue in general. Still, both subdev and the host device may support same control id. And then even though the control ids are same on the subdev and the host they could mean physically different controls (since when registering a subdev at the host driver the host's controls take precedence and doubling subdev controls are skipped).

True, but, except for specific usecases, the host control is enough.

Not for embedded devices. In many case the control won't even be implemented by the host. If your system has two sensors connected to the same host, they will both expose an exposure time control. Which one do you configure through the video node ? The two sensors will likely have different bounds for the same control, how do you report that ?

Quoted 5 lines

Quoted 3 lines

If the device has two sensors that are mutually exclusive, they should be mapped as two different inputs. The showed control should be the one used by the currently active input.

Video nodes should represent a dma engine rather than an image source.

True. Image sources are represented, on V4L2, by inputs.

Quoted 5 lines

You
could use the same hardware to process data from memory to memory while streaming video from a sensor to memory at the same time. This is a quite typical use in embedded systems.

Quoted 7 lines

Usually boards where two sensors are connected to an isp the dependencies are not as straightforward as the sensors being fully independent or require exclusive access. Typically some of the hardware blocks are shared between the two and can be used by only one sensor at a time, so instead you may not get full functionality from both at the same time. And you need to be able to choose which one uses that hardware block. This is exactly what Media controller interface models perfectly.

I see.

Quoted 11 lines

See FIMC Media controller graph AND Sylwester's explanation on it; a few links are actually missing from the grapg.

<URL:http://www.spinics.net/lists/linux-media/msg35504.html> <URL:http://wstaw.org/m/2011/05/26/fimc_graph__.png>

Cc Hans.

Quoted 3 lines

If the sensors aren't mutually exclusive, then two different video nodes will be shown in userspace.

Quoted 8 lines

Quoted 6 lines

Quoted 3 lines

Those controls are also quite useless for generic V4L2 applications, which will want auto-exposure anyway. This needs to be implemented in userspace in libv4l.

Several webcams export exposure controls. Why shouldn't those controls be exposed to userspace anymore?

This is not a webcam,

I know, but the analogy is still valid.

Quoted 5 lines

it is a software controlled high end digital camera on
a mobile device. The difference is that the functionality offered by the hardware is at much lower level compared to a webcam; the result is that more detailed control ia required but also much flexibility and performance is gained.

I see. What I failed to see is why to remove control from userspace. If the hardware is more powerful, I expect to see more controls exported, and not removing the V4L2 functionality from the driver.

Quoted 6 lines

Quoted 4 lines

Ok, if the hardware won't support 3A algorithm, libv4l will implement it, eventually using an extra hardware-aware code to get the best performance for that specific device, but this doesn't mean that the user should always use it.

Why not? What would be the alternative?

User may want or need to disable the 3A algo and control some hardware parameters hardware directly, for example, to take an over-exposed picture, to create some neat effects, or to get some image whose exposure aperture/time is out of the 3A range.

Quoted 11 lines

Quoted 5 lines

Btw, the 3A algorithm is one of the things I don't like on my cell phone: while it works most of the time, sometimes I want to disable it and manually adjust, as it produces dark images, when there's a very bright light somewhere on the image background. Manually adjusting the exposure time and aperture is something relevant for some users.

You do want the 3A algorithms even if you use manual white balance. What the automatic white balance algorithm produces is (among other things) gamma tables, rgb-to-rgb conversion matrices and lens shading correction tables. I doubt any end user, even if it was you, would like to fiddle with such large tables directly when capturing photos.

There are some hacks for several professional and amateur cameras that replace the existing 3A algorithms by... NONE. The idea is to get the raw data directly from the sensor, and use some software like Gimp or Photoshop to do lens correction, temperature correction, whitespace ballance, etc, at post-processing. The advantage of such type of usage is that the photographer can fine-tune the generated image to produce what he wants, using more sophisticated (and not real-time) algorithms.

[1] for example, one of such firmwares, that I use on my Canon Digital Camera is available at:
http://chdk.wikia.com/wiki/CHDK

Quoted 11 lines

The size of this configuration could
easily be around 10 kB. A higher level control is required; colour temperature, for instance. And its implementation involves the same automatic white balance algorithm.

You must know your hardware very, very well to use the aforementioned low level controls and in such a case you have no reason not to use the MC interface to configure the device either. Configuring the device image pipe using MC is actually a number of magnitudes less complex, and I say it's been quite an achievement that we have such an interface which makes it so effortless to do.

For sure, such kind of controls that exposes the 3A correction algorithm at the DSP level shouldn't be exposed via V4L2 interface, but things like disabling 3A and manually controlling the sensor, like aperture, exposure, analog zoom, etc, makes sense to be exported.

Quoted 35 lines

Quoted 35 lines

Quoted 28 lines

Quoted 19 lines

Quoted 3 lines

Also there might be some preference at user space, at which stage of the pipeline to apply some controls. This is where the subdev API helps, and plain video node API does not.

Again, this is for specific usecases. On such cases, what is expected is that the more generic control will be exported via V4L2 API.

Quoted 12 lines

Quoted 11 lines

Quoted 3 lines

Thus it's a bit hard to imagine that we could do something like "optionally not to inherit controls" as the subdev/MC API is optional. :)

This was actually implemented. There are some cases at ivtv/cx18 driver where both the bridge and a subdev provides the same control (audio volume, for example). The idea is to allow the bridge driver to touch at the subdev control without exposing it to userspace, since the desire was that the bridge driver itself would expose such control, using a logic that combines changing the subdev and the bridge registers for volume.

Quoted 8 lines

Quoted 3 lines

This seem like hard coding a policy in the driver;) Then there is no way (it might not be worth the effort though) to play with volume level at both devices, e.g. to obtain optimal S/N ratio.

In general, playing with just one control is enough. Andy had a different opinion when this issue were discussed, and he thinks that playing with both is better. At the end, this is a developers decision, depending on how much information (and bug reports) he had.

Quoted 4 lines

ivtv/cx18 is a completely different use case, where hardware configurations are known, and experiments possible to find out which control(s) to use and how. In this case you can't know in advance what the sensor and host drivers will be used for.

Why not? I never saw an embedded hardware that allows physically changing the sensor.

Quoted 12 lines

Quoted 11 lines

Quoted 3 lines

Even if you did, fine image quality tuning requires accessing pretty much all controls individually anyway.

The same is also true for non-embedded hardware. The only situation where V4L2 API is not enough is when there are two controls of the same type active. For example, 2 active volume controls, one at the audio demod, and another at the bridge. There may have some cases where you can do the same thing at the sensor or at a DSP block. This is where MC API gives an improvement, by allowing changing both, instead of just one of the controls.

Quoted 6 lines

This may be true on non-embedded hardware. It's important to know which hardware component implements a particular control; for example digital gain typically would take longer to have an effect if it is set on sensor rather than on the ISP. Also, is you set digital gain, you want to set it on the same device where your analog gain is --- the sensor --- to avoid badly exposed imagws.

I'd say that the driver should expose the hardware control, if 3A is disabled.

Quoted 5 lines

When it comes to scaling, the scaling quality, power consumption and performance may well be very different depending on where it is done. There typically are data rate limitations at different parts of the pipeline. The plain V4L2 has never been meant for this nor provides any support when doing something like above, and these are just few examples.

I see your point.

Quoted 64 lines

Quoted 55 lines

Quoted 55 lines

Quoted 54 lines

Quoted 5 lines

This is a hack...sorry, just joking ;-) Seriously, I think the situation with the userspace subdevs is a bit different. Because with one API we directly expose some functionality for applications, with other we code it in the kernel, to make the devices appear uniform at user space.

Not sure if I understood you. V4L2 export drivers functionality to userspace in an uniform way. MC api is for special applications that might need to access some internal functions on embedded devices.

Of course, there are some cases where it doesn't make sense to export a subdev control via V4L2 API.

Quoted 10 lines

Quoted 9 lines

Quoted 4 lines

Also, the sensor subdev can be configured in the video node driver as well as through the subdev device node. Both APIs can do the same thing but in order to let the subdev API work as expected the video node driver must be forbidden to configure the subdev.

Why? For the sensor, a V4L2 API call will look just like a bridge driver call. The subdev will need a mutex anyway, as two MC applications may be opening it simultaneously. I can't see why it should forbid changing the control from the bridge driver call.

Quoted 10 lines

Please do not forget there might be more than one subdev to configure and that the bridge itself is also a subdev (which exposes a scaler interface, for instance). A situation pretty much like in Figure 4.4 [1] (after the scaler there is also a video node to configure, but we may assume that pixel resolution at the scaler pad 1 is same as at the video node). Assuming the format and crop configuration flow is from sensor to host scaler direction, if we have tried to configure _all_ subdevs when the last stage of the pipeline is configured (i.e. video node) the whole scaler and crop/composition configuration we have been destroyed at that time. And there is more to configure than VIDIOC_S_FMT can do.

Think from users perspective: all user wants is to see a video of a given resolution. S_FMT (and a few other VIDIOC_* calls) have everything that the user wants: the desired resolution, framerate and format.

Specialized applications indeed need more, in order to get the best images for certain types of usages. So, MC is there.

Such applications will probably need to know exactly what's the sensor, what are their bugs, how it is connected, what are the DSP blocks in the patch, how the DSP algorithms are implemented, etc, in order to obtain the the perfect image.

Even on embedded devices like smartphones and tablets, I predict that both types of applications will be developed and used: people may use a generic application like flash player, and an specialized application provided by the manufacturer. Users can even develop their own applications generic apps using V4L2 directly, at the devices that allow that.

As I said before: both application types are welcome. We just need to warrant that a pure V4L application will work reasonably well.

Quoted 8 lines

Quoted 4 lines

That's why we have libv4l. The driver simply doesn't receive enough information to configure the hardware correctly from the VIDIOC_* calls. And as mentioned above, 3A algorithms, required by "simple" V4L2 applications, need to be implemented in userspace anyway.

It is OK to improve users experience via libv4l. What I'm saying is that it is NOT OK to remove V4L2 API support from the driver, forcing users to use some hardware plugin at libv4l.

Quoted 5 lines

Either do you know your hardware or do not know it. General purpose applications can rely on functionality provided by libv4l, but if you do not use it, then you need to configure the underlying device. Which is something where the Media controller and v4l2_subdev interfaces are tremendously useful.

Agreed, but while we don't actually have libv4l hw-specific plugins committed at the v4l-utils git tree, all we have to warrant that the hardware will work with a generic userspace application is the V4L2 API.

Quoted 42 lines

Quoted 42 lines

Quoted 41 lines

Quoted 31 lines

Quoted 2 lines

Allowing the bridge driver to configure subdevs at all times would prevent the subdev/MC API to work.

Well, then we need to think on an alternative for that. It seems an interesting theme for the media workshop at the Kernel Summit/2011.

Quoted 13 lines

Quoted 13 lines

Quoted 2 lines

There is a conflict there that in order to use 'optional' API the 'main' API behaviour must be affected....

It is optional from userspace perspective. A V4L2-only application should be able to work with all drivers. However, a MC-aware application will likely be specific for some hardware, as it will need to know some device-specific stuff.

Both kinds of applications are welcome, but dropping support for V4L2-only applications is the wrong thing to do.

Quoted 1 line

And I really cant use V4L2 API only as is because it's too limited.

Quoted 7 lines

Quoted 1 line

Why?

For instance there is really yet no support for scaler and composition onto a target buffer in the Video Capture Interface (we also use sensors with built in scalers). It's difficult to efficiently manage capture/preview pipelines. It is impossible to discover the system topology.

Scaler were always supported by V4L2: if the resolution specified by S_FMT is not what the sensor provides, then scale. All non-embedded drivers with sensor or bridge supports scale does that.

Quoted 7 lines

Composition is not properly supported yet. It could make sense to add it to V4L. How do you think MC API would help with composite?

Managing capture/preview pipelines will require some support at V4L2 level. This is a problem that needs to be addressed there anyway, as buffers for preview/capture need to be allocated. There's an RFC about that, but I don't think it covers the pipelines for it.

Managing pipelines is a policy decision, and needs to be implemented in userspace. Once again, the solution here is libv4l.

Quoted 7 lines

Quoted 2 lines

If V4L2 API is not enough, implementing it on libv4l won't solve, as userspace apps will use V4L2 API for requresting it.

There are two kind of applications: specialised and generic. The generic ones may rely on restrictive policies put in place by a libv4l plugin whereas the specialised applications need to access the device's features directly to get the most out of it.

A submitted upstream driver should be capable of working with the existing tools/userspace.

Currently, there isn't such libv4l plugins (or, at least, I failed to see a merged plugin there for N9, S5P, etc). Let's not upstream new drivers or remove functionalities from already existing drivers based on something that has yet to be developed.

After having it there properly working and tested independently, we may consider patches removing V4L2 interfaces that were obsoleted in favor of using the libv4l implementation, of course using the Kernel way of deprecating interfaces. But doing it before having it, doesn't make any sense.

Let's not put the the cart before the horse.

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Why not? I never saw an embedded hardware that allows physically changing the sensor.

I understood Laurent's statement that you can have same ISP driver deployed on multiple boards fitted with various sensors. Hence the multiple configurations that cannot be known in advance,

Quoted 21 lines

Quoted 7 lines

Quoted 2 lines

If V4L2 API is not enough, implementing it on libv4l won't solve, as userspace apps will use V4L2 API for requresting it.

There are two kind of applications: specialised and generic. The generic ones may rely on restrictive policies put in place by a libv4l plugin whereas the specialised applications need to access the device's features directly to get the most out of it.

A submitted upstream driver should be capable of working with the existing tools/userspace.

Currently, there isn't such libv4l plugins (or, at least, I failed to see a merged plugin there for N9, S5P, etc). Let's not upstream new drivers or remove functionalities from already existing drivers based on something that has yet to be developed.

After having it there properly working and tested independently, we may consider patches removing V4L2 interfaces that were obsoleted in favor of using the libv4l implementation, of course using the Kernel way of deprecating interfaces. But doing it before having it, doesn't make any sense.

Quoted 1 line

Let's not put the the cart before the horse.

That's a good point. My long term plan was to deprecate and remove duplicated ioctls at the driver _once_ support for regular V4L2 interface on top of MC/subdev API is added at the v4l2 libraries. But this will happen after I create an initial.. *cough* openmax IL for the driver. Which is not what the Tigers like best..

--
Regards,
Sylwester

Em 09-08-2011 20:18, Sakari Ailus escreveu:

Quoted 5 lines · Mauro Carvalho Chehab

Quoted 4 lines

Em 29-07-2011 05:36, Laurent Pinchart escreveu:

Quoted 3 lines · On Friday 29 July 2011 06:02:54 Mauro Carvalho Chehab wrote:

Quoted 2 lines

Em 28-07-2011 19:57, Sylwester Nawrocki escreveu:

Quoted 1 line · Mauro Carvalho Chehab

Quoted 8 lines

Quoted 6 lines

Quoted 3 lines

Those controls are also quite useless for generic V4L2 applications, which will want auto-exposure anyway. This needs to be implemented in userspace in libv4l.

Several webcams export exposure controls. Why shouldn't those controls be exposed to userspace anymore?

This is not a webcam,

I know, but the analogy is still valid.

Quoted 5 lines

it is a software controlled high end digital camera on
a mobile device. The difference is that the functionality offered by the hardware is at much lower level compared to a webcam; the result is that more detailed control ia required but also much flexibility and performance is gained.

I see. What I failed to see is why to remove control from userspace. If the hardware is more powerful, I expect to see more controls exported, and not removing the V4L2 functionality from the driver.

Quoted 6 lines

Quoted 4 lines

Ok, if the hardware won't support 3A algorithm, libv4l will implement it, eventually using an extra hardware-aware code to get the best performance for that specific device, but this doesn't mean that the user should always use it.

Why not? What would be the alternative?

User may want or need to disable the 3A algo and control some hardware parameters hardware directly, for example, to take an over-exposed picture, to create some neat effects, or to get some image whose exposure aperture/time is out of the 3A range.

Quoted 11 lines

Quoted 5 lines

Btw, the 3A algorithm is one of the things I don't like on my cell
phone: while it works most of the time, sometimes I want to disable it and manually adjust, as it produces dark images, when there's a very bright light somewhere on the image background. Manually adjusting the exposure time and aperture is something relevant for some users.

You do want the 3A algorithms even if you use manual white balance. What the automatic white balance algorithm produces is (among other things) gamma tables, rgb-to-rgb conversion matrices and lens shading correction tables. I doubt any end user, even if it was you, would like to fiddle with such large tables directly when capturing photos.

There are some hacks for several professional and amateur cameras that replace the existing 3A algorithms by... NONE. The idea is to get the raw data directly from the sensor, and use some software like Gimp or Photoshop to do lens correction, temperature correction, whitespace ballance, etc, at post-processing. The advantage of such type of usage is that the photographer can fine-tune the generated image to produce what he wants, using more sophisticated (and not real-time) algorithms.

[1] for example, one of such firmwares, that I use on my Canon Digital Camera is available at:
http://chdk.wikia.com/wiki/CHDK

Quoted 11 lines

The size of this configuration could
easily be around 10 kB. A higher level control is required; colour temperature, for instance. And its implementation involves the same automatic white balance algorithm.

You must know your hardware very, very well to use the aforementioned low level controls and in such a case you have no reason not to use the MC interface to configure the device either. Configuring the device image pipe using MC is actually a number of magnitudes less complex, and I say it's been quite an achievement that we have such an interface which makes it so effortless to do.

For sure, such kind of controls that exposes the 3A correction algorithm at the DSP level shouldn't be exposed via V4L2 interface, but things like disabling 3A and manually controlling the sensor, like aperture, exposure, analog zoom, etc, makes sense to be exported.

Quoted 49 lines

Quoted 41 lines

Quoted 30 lines

Quoted 21 lines

Quoted 3 lines

Also there might be some preference at user space, at which stage of the pipeline to apply some controls. This is where the subdev API helps, and plain video node API does not.

Again, this is for specific usecases. On such cases, what is expected is that the more generic control will be exported via V4L2 API.

Quoted 14 lines

Quoted 13 lines

Quoted 5 lines

Thus it's a bit hard to imagine that we could do something like "optionally not to inherit controls" as the subdev/MC API is optional.

:)

This was actually implemented. There are some cases at ivtv/cx18 driver where both the bridge and a subdev provides the same control (audio volume, for example). The idea is to allow the bridge driver to touch at the subdev control without exposing it to userspace, since the desire was that the bridge driver itself would expose such control, using a logic that combines changing the subdev and the bridge registers for volume.

Quoted 8 lines

Quoted 3 lines

This seem like hard coding a policy in the driver;) Then there is no way (it might not be worth the effort though) to play with volume level at both devices, e.g. to obtain optimal S/N ratio.

In general, playing with just one control is enough. Andy had a different opinion when this issue were discussed, and he thinks that playing with both is better. At the end, this is a developers decision, depending on how much information (and bug reports) he had.

Quoted 4 lines

ivtv/cx18 is a completely different use case, where hardware configurations are known, and experiments possible to find out which control(s) to use and how. In this case you can't know in advance what the sensor and host drivers will be used for.

Why not? I never saw an embedded hardware that allows physically changing the sensor.

Quoted 2 lines

Even if you did, fine image quality tuning requires accessing pretty much all controls individually anyway.

Quoted 7 lines

The same is also true for non-embedded hardware. The only situation where V4L2 API is not enough is when there are two controls of the same type active. For example, 2 active volume controls, one at the audio demod, and another at the bridge. There may have some cases where you can do the same thing at the sensor or at a DSP block. This is where MC API gives an improvement, by allowing changing both, instead of just one of the controls.

Quoted 6 lines

This may be true on non-embedded hardware. It's important to know which hardware component implements a particular control; for example digital gain typically would take longer to have an effect if it is set on sensor rather than on the ISP. Also, is you set digital gain, you want to set it on the same device where your analog gain is --- the sensor --- to avoid badly exposed imagws.

I'd say that the driver should expose the hardware control, if 3A is disabled.

Quoted 6 lines

When it comes to scaling, the scaling quality, power consumption and performance may well be very different depending on where it is done. There typically are data rate limitations at different parts of the pipeline. The plain V4L2 has never been meant for this nor provides any support when doing something like above, and these are just few examples.

I see your point.

Quoted 67 lines

Quoted 58 lines

Quoted 58 lines

Quoted 57 lines

Quoted 5 lines

This is a hack...sorry, just joking ;-) Seriously, I think the situation with the userspace subdevs is a bit different. Because with one API we directly expose some functionality for applications, with other we code it in the kernel, to make the devices appear uniform at user space.

Not sure if I understood you. V4L2 export drivers functionality to userspace in an uniform way. MC api is for special applications that might need to access some internal functions on embedded devices.

Of course, there are some cases where it doesn't make sense to export a subdev control via V4L2 API.

Quoted 10 lines

Quoted 9 lines

Quoted 4 lines

Also, the sensor subdev can be configured in the video node driver as well as through the subdev device node. Both APIs can do the same thing but in order to let the subdev API work as expected the video node driver must be forbidden to configure the subdev.

Why? For the sensor, a V4L2 API call will look just like a bridge driver call. The subdev will need a mutex anyway, as two MC applications may be opening it simultaneously. I can't see why it should forbid changing the control from the bridge driver call.

Quoted 11 lines

Please do not forget there might be more than one subdev to configure and that the bridge itself is also a subdev (which exposes a scaler interface, for instance). A situation pretty much like in Figure 4.4
[1] (after the scaler there is also a video node to configure, but we may assume that pixel resolution at the scaler pad 1 is same as at the video node). Assuming the format and crop configuration flow is from sensor to host scaler direction, if we have tried to configure _all_ subdevs when the last stage of the pipeline is configured (i.e. video node) the whole scaler and crop/composition configuration we have been destroyed at that time. And there is more to configure than VIDIOC_S_FMT can do.

Think from users perspective: all user wants is to see a video of a given resolution. S_FMT (and a few other VIDIOC_* calls) have everything that the user wants: the desired resolution, framerate and format.

Specialized applications indeed need more, in order to get the best images for certain types of usages. So, MC is there.

Such applications will probably need to know exactly what's the sensor, what are their bugs, how it is connected, what are the DSP blocks in the patch, how the DSP algorithms are implemented, etc, in order to obtain the the perfect image.

Even on embedded devices like smartphones and tablets, I predict that both types of applications will be developed and used: people may use a generic application like flash player, and an specialized application provided by the manufacturer. Users can even develop their own applications generic apps using V4L2 directly, at the devices that allow that.

As I said before: both application types are welcome. We just need to warrant that a pure V4L application will work reasonably well.

Quoted 8 lines

Quoted 4 lines

That's why we have libv4l. The driver simply doesn't receive enough information to configure the hardware correctly from the VIDIOC_* calls. And as mentioned above, 3A algorithms, required by "simple" V4L2 applications, need to be implemented in userspace anyway.

It is OK to improve users experience via libv4l. What I'm saying is that it is NOT OK to remove V4L2 API support from the driver, forcing users to use some hardware plugin at libv4l.

Quoted 5 lines

Either do you know your hardware or do not know it. General purpose applications can rely on functionality provided by libv4l, but if you do not use it, then you need to configure the underlying device. Which is something where the Media controller and v4l2_subdev interfaces are tremendously useful.

Agreed, but while we don't actually have libv4l hw-specific plugins committed at the v4l-utils git tree, all we have to warrant that the hardware will work with a generic userspace application is the V4L2 API.

Quoted 42 lines

Quoted 42 lines

Quoted 41 lines

Quoted 31 lines

Quoted 2 lines

Allowing the bridge driver to configure subdevs at all times would prevent the subdev/MC API to work.

Well, then we need to think on an alternative for that. It seems an interesting theme for the media workshop at the Kernel Summit/2011.

Quoted 13 lines

Quoted 13 lines

Quoted 2 lines

There is a conflict there that in order to use 'optional' API the 'main' API behaviour must be affected....

It is optional from userspace perspective. A V4L2-only application should be able to work with all drivers. However, a MC-aware application will likely be specific for some hardware, as it will need to know some device-specific stuff.

Both kinds of applications are welcome, but dropping support for V4L2-only applications is the wrong thing to do.

Quoted 1 line

And I really cant use V4L2 API only as is because it's too limited.

Quoted 7 lines

Quoted 1 line

Why?

For instance there is really yet no support for scaler and composition onto a target buffer in the Video Capture Interface (we also use sensors with built in scalers). It's difficult to efficiently manage capture/preview pipelines. It is impossible to discover the system topology.

Scaler were always supported by V4L2: if the resolution specified by S_FMT is not what the sensor provides, then scale. All non-embedded drivers with sensor or bridge supports scale does that.

Quoted 7 lines

Composition is not properly supported yet. It could make sense to add it to V4L. How do you think MC API would help with composite?

Managing capture/preview pipelines will require some support at V4L2 level. This is a problem that needs to be addressed there anyway, as buffers for preview/capture need to be allocated. There's an RFC about that, but I don't think it covers the pipelines for it.

Managing pipelines is a policy decision, and needs to be implemented in userspace. Once again, the solution here is libv4l.

Quoted 7 lines

Quoted 2 lines

If V4L2 API is not enough, implementing it on libv4l won't solve, as userspace apps will use V4L2 API for requresting it.

There are two kind of applications: specialised and generic. The generic ones may rely on restrictive policies put in place by a libv4l plugin whereas the specialised applications need to access the device's features directly to get the most out of it.

A submitted upstream driver should be capable of working with the existing tools/userspace.

Currently, there isn't such libv4l plugins (or, at least, I failed to see a merged plugin there for N9, S5P, etc). Let's not upstream new drivers or remove functionalities from already existing drivers based on something that has yet to be developed.

After having it there properly working and tested independently, we may consider patches removing V4L2 interfaces that were obsoleted in favor of using the libv4l implementation, of course using the Kernel way of deprecating interfaces. But doing it before having it, doesn't make any sense.

Let's not put the the cart before the horse.

Hi Mauro,

Quoted 5 lines

After having it there properly working and tested independently, we may consider patches removing V4L2 interfaces that were obsoleted in favor of using the libv4l implementation, of course using the Kernel way of deprecating interfaces. But doing it before having it, doesn't make any sense.

Once again we're not trying to remove controls, but expose them differently.

Em 15-08-2011 05:45, Laurent Pinchart escreveu:

Quoted 6 lines

Quoted 5 lines

After having it there properly working and tested independently, we may consider patches removing V4L2 interfaces that were obsoleted in favor of using the libv4l implementation, of course using the Kernel way of deprecating interfaces. But doing it before having it, doesn't make any sense.

Quoted 2 lines

Once again we're not trying to remove controls, but expose them differently.

See the comments at the patch series, starting from the patches that removes support for S_INPUT. I'm aware that the controls will be exposed by both MC and V4L2 API, althrough having ways to expose/hide controls via V4L2 API makes patch review a way more complicated than it used to be before the MC patches.

Hi Mauro,

Quoted 7 lines · On Tuesday 16 August 2011 02:21:09 Mauro Carvalho Chehab wrote:

Em 15-08-2011 05:45, Laurent Pinchart escreveu:

Quoted 6 lines

Quoted 5 lines

After having it there properly working and tested independently, we may consider patches removing V4L2 interfaces that were obsoleted in favor of using the libv4l implementation, of course using the Kernel way of deprecating interfaces. But doing it before having it, doesn't make any sense.

Quoted 8 lines

Quoted 2 lines

Once again we're not trying to remove controls, but expose them differently.

See the comments at the patch series, starting from the patches that removes support for S_INPUT. I'm aware that the controls will be exposed by both MC and V4L2 API, althrough having ways to expose/hide controls via V4L2 API makes patch review a way more complicated than it used to be before the MC patches.

The MC API doesn't expose controls. Controls are still exposed by the V4L2 API only, but V4L2 can then expose them on subdev nodes in addition to video nodes.

Em 29-07-2011 05:36, Laurent Pinchart escreveu:

Quoted 19 lines · On Friday 29 July 2011 06:02:54 Mauro Carvalho Chehab wrote:

Quoted 18 lines

Em 28-07-2011 19:57, Sylwester Nawrocki escreveu:

Quoted 16 lines · Mauro Carvalho Chehab

Quoted 8 lines

Accumulating sub-dev controls at the video node is the right thing to do.

An MC-aware application will need to handle with that, but that doesn't sound to be hard. All such application would need to do is to first probe the subdev controls, and, when parsing the videodev controls, not register controls with duplicated ID's, or to mark them with some special attribute.

IMHO it's not a big issue in general. Still, both subdev and the host device may support same control id. And then even though the control ids are same on the subdev and the host they could mean physically different controls (since when registering a subdev at the host driver the host's controls take precedence and doubling subdev controls are skipped).

Quoted 7 lines

Quoted 1 line

True, but, except for specific usecases, the host control is enough.

Not for embedded devices. In many case the control won't even be implemented by the host. If your system has two sensors connected to the same host, they will both expose an exposure time control. Which one do you configure through the video node ? The two sensors will likely have different bounds for the same control, how do you report that ?

If the device has two sensors that are mutually exclusive, they should be mapped as two different inputs. The showed control should be the one used by the currently active input.

If the sensors aren't mutually exclusive, then two different video nodes will be shown in userspace.

Quoted 3 lines

Those controls are also quite useless for generic V4L2 applications, which will want auto-exposure anyway. This needs to be implemented in userspace in libv4l.

Several webcams export exposure controls. Why shouldn't those controls be exposed to userspace anymore?

Ok, if the hardware won't support 3A algorithm, libv4l will implement it, eventually using an extra hardware-aware code to get the best performance for that specific device, but this doesn't mean that the user should always use it.

Btw, the 3A algorithm is one of the things I don't like on my cell phone:
while it works most of the time, sometimes I want to disable it and manually adjust, as it produces dark images, when there's a very bright light somewhere on the image background. Manually adjusting the exposure time and aperture is something relevant for some users.

Quoted 30 lines

Quoted 29 lines

Quoted 3 lines

Also there might be some preference at user space, at which stage of the pipeline to apply some controls. This is where the subdev API helps, and plain video node API does not.

Again, this is for specific usecases. On such cases, what is expected is that the more generic control will be exported via V4L2 API.

Quoted 14 lines

Quoted 6 lines

Quoted 5 lines

Thus it's a bit hard to imagine that we could do something like "optionally not to inherit controls" as the subdev/MC API is optional.

:)

Quoted 7 lines

This was actually implemented. There are some cases at ivtv/cx18 driver where both the bridge and a subdev provides the same control (audio volume, for example). The idea is to allow the bridge driver to touch at the subdev control without exposing it to userspace, since the desire was that the bridge driver itself would expose such control, using a logic that combines changing the subdev and the bridge registers for volume.

Quoted 3 lines

This seem like hard coding a policy in the driver;) Then there is no way (it might not be worth the effort though) to play with volume level at both devices, e.g. to obtain optimal S/N ratio.

In general, playing with just one control is enough. Andy had a different opinion when this issue were discussed, and he thinks that playing with both is better. At the end, this is a developers decision, depending on how much information (and bug reports) he had.

Quoted 4 lines

ivtv/cx18 is a completely different use case, where hardware configurations are known, and experiments possible to find out which control(s) to use and how. In this case you can't know in advance what the sensor and host drivers will be used for.

Why not?

I never saw an embedded hardware that allows physically changing the sensor.

Quoted 2 lines

Even if you did, fine image quality tuning requires accessing pretty much all controls individually anyway.

The same is also true for non-embedded hardware. The only situation where V4L2 API is not enough is when there are two controls of the same type active. For example, 2 active volume controls, one at the audio demod, and another at the bridge. There may have some cases where you can do the same thing at the sensor or at a DSP block. This is where MC API gives an improvement, by allowing changing both, instead of just one of the controls.

Quoted 58 lines

Quoted 57 lines

Quoted 5 lines

This is a hack...sorry, just joking ;-) Seriously, I think the situation with the userspace subdevs is a bit different. Because with one API we directly expose some functionality for applications, with other we code it in the kernel, to make the devices appear uniform at user space.

Not sure if I understood you. V4L2 export drivers functionality to userspace in an uniform way. MC api is for special applications that might need to access some internal functions on embedded devices.

Of course, there are some cases where it doesn't make sense to export a subdev control via V4L2 API.

Quoted 10 lines

Quoted 9 lines

Quoted 4 lines

Also, the sensor subdev can be configured in the video node driver as well as through the subdev device node. Both APIs can do the same thing but in order to let the subdev API work as expected the video node driver must be forbidden to configure the subdev.

Why? For the sensor, a V4L2 API call will look just like a bridge driver call. The subdev will need a mutex anyway, as two MC applications may be opening it simultaneously. I can't see why it should forbid changing the control from the bridge driver call.

Quoted 11 lines

Please do not forget there might be more than one subdev to configure and that the bridge itself is also a subdev (which exposes a scaler interface, for instance). A situation pretty much like in Figure 4.4
[1] (after the scaler there is also a video node to configure, but we may assume that pixel resolution at the scaler pad 1 is same as at the video node). Assuming the format and crop configuration flow is from sensor to host scaler direction, if we have tried to configure _all_ subdevs when the last stage of the pipeline is configured (i.e. video node) the whole scaler and crop/composition configuration we have been destroyed at that time. And there is more to configure than VIDIOC_S_FMT can do.

Think from users perspective: all user wants is to see a video of a given resolution. S_FMT (and a few other VIDIOC_* calls) have everything that the user wants: the desired resolution, framerate and format.

Specialized applications indeed need more, in order to get the best images for certain types of usages. So, MC is there.

Such applications will probably need to know exactly what's the sensor, what are their bugs, how it is connected, what are the DSP blocks in the patch, how the DSP algorithms are implemented, etc, in order to obtain the the perfect image.

Even on embedded devices like smartphones and tablets, I predict that both types of applications will be developed and used: people may use a generic application like flash player, and an specialized application provided by the manufacturer. Users can even develop their own applications generic apps using V4L2 directly, at the devices that allow that.

As I said before: both application types are welcome. We just need to warrant that a pure V4L application will work reasonably well.

Quoted 4 lines

That's why we have libv4l. The driver simply doesn't receive enough information to configure the hardware correctly from the VIDIOC_* calls. And as mentioned above, 3A algorithms, required by "simple" V4L2 applications, need to be implemented in userspace anyway.

It is OK to improve users experience via libv4l. What I'm saying is that it is NOT OK to remove V4L2 API support from the driver, forcing users to use some hardware plugin at libv4l.

Quoted 39 lines

Quoted 38 lines

Quoted 2 lines

Allowing the bridge driver to configure subdevs at all times would prevent the subdev/MC API to work.

Well, then we need to think on an alternative for that. It seems an interesting theme for the media workshop at the Kernel Summit/2011.

Quoted 13 lines

Quoted 13 lines

Quoted 2 lines

There is a conflict there that in order to use 'optional' API the 'main' API behaviour must be affected....

It is optional from userspace perspective. A V4L2-only application should be able to work with all drivers. However, a MC-aware application will likely be specific for some hardware, as it will need to know some device-specific stuff.

Both kinds of applications are welcome, but dropping support for V4L2-only applications is the wrong thing to do.

Quoted 1 line

And I really cant use V4L2 API only as is because it's too limited.

Quoted 7 lines

Quoted 1 line

Why?

For instance there is really yet no support for scaler and composition onto a target buffer in the Video Capture Interface (we also use sensors with built in scalers). It's difficult to efficiently manage capture/preview pipelines. It is impossible to discover the system topology.

Scaler were always supported by V4L2: if the resolution specified by S_FMT is not what the sensor provides, then scale. All non-embedded drivers with sensor or bridge supports scale does that.

Composition is not properly supported yet. It could make sense to add it to V4L. How do you think MC API would help with composite?

Managing capture/preview pipelines will require some support at V4L2 level. This is a problem that needs to be addressed there anyway, as buffers for preview/capture need to be allocated. There's an RFC about that, but I don't think it covers the pipelines for it.

Quoted 2 lines

Managing pipelines is a policy decision, and needs to be implemented in userspace. Once again, the solution here is libv4l.

If V4L2 API is not enough, implementing it on libv4l won't solve, as userspace apps will use V4L2 API for requresting it.

Hi Mauro,

Quoted 32 lines · On Tuesday 09 August 2011 22:05:47 Mauro Carvalho Chehab wrote:

Em 29-07-2011 05:36, Laurent Pinchart escreveu:

Quoted 19 lines · On Friday 29 July 2011 06:02:54 Mauro Carvalho Chehab wrote:

Quoted 18 lines

Em 28-07-2011 19:57, Sylwester Nawrocki escreveu:

Quoted 16 lines · Mauro Carvalho Chehab

Quoted 8 lines

Accumulating sub-dev controls at the video node is the right thing to do.

An MC-aware application will need to handle with that, but that doesn't sound to be hard. All such application would need to do is to first probe the subdev controls, and, when parsing the videodev controls, not register controls with duplicated ID's, or to mark them with some special attribute.

IMHO it's not a big issue in general. Still, both subdev and the host device may support same control id. And then even though the control ids are same on the subdev and the host they could mean physically different controls (since when registering a subdev at the host driver the host's controls take precedence and doubling subdev controls are skipped).

Quoted 7 lines

Quoted 1 line

True, but, except for specific usecases, the host control is enough.

Not for embedded devices. In many case the control won't even be implemented by the host. If your system has two sensors connected to the same host, they will both expose an exposure time control. Which one do you configure through the video node ? The two sensors will likely have different bounds for the same control, how do you report that ?

If the device has two sensors that are mutually exclusive, they should be mapped as two different inputs. The showed control should be the one used by the currently active input.

Quoted 2 lines

If the sensors aren't mutually exclusive, then two different video nodes will be shown in userspace.

It's more complex than that. The OMAP3 ISP driver exposes 7 video nodes regardless of the number of sensors. Sensors can be mutually exclusive or not, depending on the board. S_INPUT has its use cases, but is less useful on embedded hardware.

Quoted 12 lines

Quoted 3 lines

Those controls are also quite useless for generic V4L2 applications, which will want auto-exposure anyway. This needs to be implemented in userspace in libv4l.

Several webcams export exposure controls. Why shouldn't those controls be exposed to userspace anymore?

Ok, if the hardware won't support 3A algorithm, libv4l will implement it, eventually using an extra hardware-aware code to get the best performance for that specific device, but this doesn't mean that the user should always use it.

Quoted 5 lines

Btw, the 3A algorithm is one of the things I don't like on my cell phone:
while it works most of the time, sometimes I want to disable it and manually adjust, as it produces dark images, when there's a very bright light somewhere on the image background. Manually adjusting the exposure time and aperture is something relevant for some users.

It is, but on embedded devices that usually requires the application to be hardware-aware. Exposure time limits depend on blanking, which in turn influences the frame rate along with the pixel clock (often configurable as well). Programming those settings wrong can exceed the ISP available bandwidth.

The world unfortunately stopped being simple some time ago :-)

Quoted 35 lines

Quoted 34 lines

Quoted 21 lines

Quoted 3 lines

Also there might be some preference at user space, at which stage of the pipeline to apply some controls. This is where the subdev API helps, and plain video node API does not.

Again, this is for specific usecases. On such cases, what is expected is that the more generic control will be exported via V4L2 API.

Quoted 14 lines

Quoted 13 lines

Quoted 5 lines

Thus it's a bit hard to imagine that we could do something like "optionally not to inherit controls" as the subdev/MC API is optional.

:)

This was actually implemented. There are some cases at ivtv/cx18 driver where both the bridge and a subdev provides the same control (audio volume, for example). The idea is to allow the bridge driver to touch at the subdev control without exposing it to userspace, since the desire was that the bridge driver itself would expose such control, using a logic that combines changing the subdev and the bridge registers for volume.

Quoted 8 lines

Quoted 3 lines

This seem like hard coding a policy in the driver;) Then there is no way (it might not be worth the effort though) to play with volume level at both devices, e.g. to obtain optimal S/N ratio.

In general, playing with just one control is enough. Andy had a different opinion when this issue were discussed, and he thinks that playing with both is better. At the end, this is a developers decision, depending on how much information (and bug reports) he had.

ivtv/cx18 is a completely different use case, where hardware configurations are known, and experiments possible to find out which control(s) to use and how. In this case you can't know in advance what the sensor and host drivers will be used for.

Quoted 1 line

Why not?

My point is that the ISP driver developer can't know in advance which sensor will be used systems that don't exist yet.

Quoted 2 lines

I never saw an embedded hardware that allows physically changing the sensor.

Beagleboard + pluggable sensor board.

Quoted 10 lines

Quoted 2 lines

Even if you did, fine image quality tuning requires accessing pretty much all controls individually anyway.

The same is also true for non-embedded hardware. The only situation where V4L2 API is not enough is when there are two controls of the same type active. For example, 2 active volume controls, one at the audio demod, and another at the bridge. There may have some cases where you can do the same thing at the sensor or at a DSP block. This is where MC API gives an improvement, by allowing changing both, instead of just one of the controls.

To be precise it's the V4L2 subdev userspace API that allows that, not the MC API.

Quoted 63 lines

Quoted 62 lines

Quoted 55 lines

Quoted 5 lines

This is a hack...sorry, just joking ;-) Seriously, I think the situation with the userspace subdevs is a bit different. Because with one API we directly expose some functionality for applications, with other we code it in the kernel, to make the devices appear uniform at user space.

Not sure if I understood you. V4L2 export drivers functionality to userspace in an uniform way. MC api is for special applications that might need to access some internal functions on embedded devices.

Of course, there are some cases where it doesn't make sense to export a subdev control via V4L2 API.

Quoted 10 lines

Quoted 9 lines

Quoted 4 lines

Also, the sensor subdev can be configured in the video node driver as well as through the subdev device node. Both APIs can do the same thing but in order to let the subdev API work as expected the video node driver must be forbidden to configure the subdev.

Why? For the sensor, a V4L2 API call will look just like a bridge driver call. The subdev will need a mutex anyway, as two MC applications may be opening it simultaneously. I can't see why it should forbid changing the control from the bridge driver call.

Quoted 11 lines

Please do not forget there might be more than one subdev to configure and that the bridge itself is also a subdev (which exposes a scaler interface, for instance). A situation pretty much like in Figure 4.4
[1] (after the scaler there is also a video node to configure, but we may assume that pixel resolution at the scaler pad 1 is same as at the video node). Assuming the format and crop configuration flow is from sensor to host scaler direction, if we have tried to configure _all_ subdevs when the last stage of the pipeline is configured (i.e. video node) the whole scaler and crop/composition configuration we have been destroyed at that time. And there is more to configure than VIDIOC_S_FMT can do.

Think from users perspective: all user wants is to see a video of a given resolution. S_FMT (and a few other VIDIOC_* calls) have everything that the user wants: the desired resolution, framerate and format.

Specialized applications indeed need more, in order to get the best images for certain types of usages. So, MC is there.

Such applications will probably need to know exactly what's the sensor, what are their bugs, how it is connected, what are the DSP blocks in the patch, how the DSP algorithms are implemented, etc, in order to obtain the the perfect image.

Even on embedded devices like smartphones and tablets, I predict that both types of applications will be developed and used: people may use a generic application like flash player, and an specialized application provided by the manufacturer. Users can even develop their own applications generic apps using V4L2 directly, at the devices that allow that.

Quoted 2 lines

As I said before: both application types are welcome. We just need to warrant that a pure V4L application will work reasonably well.

That's why we have libv4l. The driver simply doesn't receive enough information to configure the hardware correctly from the VIDIOC_* calls. And as mentioned above, 3A algorithms, required by "simple" V4L2 applications, need to be implemented in userspace anyway.

Quoted 3 lines

It is OK to improve users experience via libv4l. What I'm saying is that it is NOT OK to remove V4L2 API support from the driver, forcing users to use some hardware plugin at libv4l.

Let me be clear on this. I'm *NOT* advocating removing V4L2 API support from any driver (well, on the drivers I can currently think of, if you show me a wifi driver that implements a V4L2 interface I might change my mind :-)).

The V4L2 API has been designed mostly for desktop hardware. Thanks to its clean design we can use it for embedded hardware, even though it requires extensions. What we need to do is to define which parts of the whole API apply as-is to embedded hardware, and which should better be left unused.

Quoted 42 lines

Quoted 41 lines

Quoted 31 lines

Quoted 2 lines

Allowing the bridge driver to configure subdevs at all times would prevent the subdev/MC API to work.

Well, then we need to think on an alternative for that. It seems an interesting theme for the media workshop at the Kernel Summit/2011.

Quoted 13 lines

Quoted 13 lines

Quoted 2 lines

There is a conflict there that in order to use 'optional' API the 'main' API behaviour must be affected....

It is optional from userspace perspective. A V4L2-only application should be able to work with all drivers. However, a MC-aware application will likely be specific for some hardware, as it will need to know some device-specific stuff.

Both kinds of applications are welcome, but dropping support for V4L2-only applications is the wrong thing to do.

Quoted 1 line

And I really cant use V4L2 API only as is because it's too limited.

Quoted 7 lines

Quoted 1 line

Why?

For instance there is really yet no support for scaler and composition onto a target buffer in the Video Capture Interface (we also use sensors with built in scalers). It's difficult to efficiently manage capture/preview pipelines. It is impossible to discover the system topology.

Scaler were always supported by V4L2: if the resolution specified by S_FMT is not what the sensor provides, then scale. All non-embedded drivers with sensor or bridge supports scale does that.

Quoted 7 lines

Composition is not properly supported yet. It could make sense to add it to V4L. How do you think MC API would help with composite?

Managing capture/preview pipelines will require some support at V4L2 level. This is a problem that needs to be addressed there anyway, as buffers for preview/capture need to be allocated. There's an RFC about that, but I don't think it covers the pipelines for it.

Managing pipelines is a policy decision, and needs to be implemented in userspace. Once again, the solution here is libv4l.

Quoted 2 lines

If V4L2 API is not enough, implementing it on libv4l won't solve, as userspace apps will use V4L2 API for requresting it.

We need to let userspace configure the pipeline. That's what the MC + V4L2 APIs are for. The driver must no make policy decisions though, that must be left to userspace.

Em 15-08-2011 09:30, Laurent Pinchart escreveu:

Quoted 37 lines · On Tuesday 09 August 2011 22:05:47 Mauro Carvalho Chehab wrote:

Quoted 35 lines

Em 29-07-2011 05:36, Laurent Pinchart escreveu:

Quoted 19 lines · On Friday 29 July 2011 06:02:54 Mauro Carvalho Chehab wrote:

Quoted 18 lines

Em 28-07-2011 19:57, Sylwester Nawrocki escreveu:

Quoted 16 lines · Mauro Carvalho Chehab

Quoted 8 lines

Accumulating sub-dev controls at the video node is the right thing to do.

An MC-aware application will need to handle with that, but that doesn't sound to be hard. All such application would need to do is to first probe the subdev controls, and, when parsing the videodev controls, not register controls with duplicated ID's, or to mark them with some special attribute.

IMHO it's not a big issue in general. Still, both subdev and the host device may support same control id. And then even though the control ids are same on the subdev and the host they could mean physically different controls (since when registering a subdev at the host driver the host's controls take precedence and doubling subdev controls are skipped).

Quoted 8 lines

Quoted 1 line

True, but, except for specific usecases, the host control is enough.

Not for embedded devices. In many case the control won't even be implemented by the host. If your system has two sensors connected to the same host, they will both expose an exposure time control. Which one do you configure through the video node ? The two sensors will likely have different bounds for the same control, how do you report that ?

If the device has two sensors that are mutually exclusive, they should be mapped as two different inputs. The showed control should be the one used by the currently active input.

If the sensors aren't mutually exclusive, then two different video nodes will be shown in userspace.

Quoted 4 lines

It's more complex than that. The OMAP3 ISP driver exposes 7 video nodes regardless of the number of sensors. Sensors can be mutually exclusive or not, depending on the board. S_INPUT has its use cases, but is less useful on embedded hardware.

Sorry, but exposing a video node that can't be used doesn't make sense.

Quoted 61 lines

Quoted 12 lines

Quoted 3 lines

Those controls are also quite useless for generic V4L2 applications, which will want auto-exposure anyway. This needs to be implemented in userspace in libv4l.

Several webcams export exposure controls. Why shouldn't those controls be exposed to userspace anymore?

Ok, if the hardware won't support 3A algorithm, libv4l will implement it, eventually using an extra hardware-aware code to get the best performance for that specific device, but this doesn't mean that the user should always use it.

Quoted 5 lines

Btw, the 3A algorithm is one of the things I don't like on my cell
phone: while it works most of the time, sometimes I want to disable it and manually adjust, as it produces dark images, when there's a very bright light somewhere on the image background. Manually adjusting the exposure time and aperture is something relevant for some users.

It is, but on embedded devices that usually requires the application to be hardware-aware. Exposure time limits depend on blanking, which in turn influences the frame rate along with the pixel clock (often configurable as well). Programming those settings wrong can exceed the ISP available bandwidth.

The world unfortunately stopped being simple some time ago :-)

Quoted 35 lines

Quoted 34 lines

Quoted 21 lines

Quoted 3 lines

Also there might be some preference at user space, at which stage of the pipeline to apply some controls. This is where the subdev API helps, and plain video node API does not.

Again, this is for specific usecases. On such cases, what is expected is that the more generic control will be exported via V4L2 API.

Quoted 14 lines

Quoted 13 lines

Quoted 5 lines

Thus it's a bit hard to imagine that we could do something like "optionally not to inherit controls" as the subdev/MC API is optional.

:)

This was actually implemented. There are some cases at ivtv/cx18 driver where both the bridge and a subdev provides the same control (audio volume, for example). The idea is to allow the bridge driver to touch at the subdev control without exposing it to userspace, since the desire was that the bridge driver itself would expose such control, using a logic that combines changing the subdev and the bridge registers for volume.

Quoted 8 lines

Quoted 3 lines

This seem like hard coding a policy in the driver;) Then there is no way (it might not be worth the effort though) to play with volume level at both devices, e.g. to obtain optimal S/N ratio.

In general, playing with just one control is enough. Andy had a different opinion when this issue were discussed, and he thinks that playing with both is better. At the end, this is a developers decision, depending on how much information (and bug reports) he had.

ivtv/cx18 is a completely different use case, where hardware configurations are known, and experiments possible to find out which control(s) to use and how. In this case you can't know in advance what the sensor and host drivers will be used for.

Quoted 4 lines

Quoted 1 line

Why not?

My point is that the ISP driver developer can't know in advance which sensor will be used systems that don't exist yet.

As far as such hardware is projected, someone will know. It is a simple trivial patch to associate a new hardware with a hardware profile at the platform data.

Also, on most cases, probing a sensor is as trivial as reading a sensor ID during device probe. This applies, for example, for all Omnivision sensors.

We do things like that all the times for PC world, as nobody knows what webcam someone would plug on his PC.

Quoted 4 lines

Quoted 2 lines

I never saw an embedded hardware that allows physically changing the sensor.

Beagleboard + pluggable sensor board.

Development systems like beagleboard, pandaboard, Exynos SMDK, etc, aren't embeeded hardware. They're development kits.

I don't mind if, for those kits the developer that is playing with it has to pass a mode parameter and/or run some open harware-aware small application that makes the driver to select the sensor type he is using, but, if the hardware is, instead, a N9 or a Galaxy Tab (or whatever embedded hardware), the driver should expose just the sensors that exists on such hardware. It shouldn't be ever allowed to change it on userspace, using whatever API on those hardware.

Quoted 82 lines

Quoted 10 lines

Quoted 2 lines

Even if you did, fine image quality tuning requires accessing pretty much all controls individually anyway.

The same is also true for non-embedded hardware. The only situation where V4L2 API is not enough is when there are two controls of the same type active. For example, 2 active volume controls, one at the audio demod, and another at the bridge. There may have some cases where you can do the same thing at the sensor or at a DSP block. This is where MC API gives an improvement, by allowing changing both, instead of just one of the controls.

To be precise it's the V4L2 subdev userspace API that allows that, not the MC API.

Quoted 67 lines

Quoted 56 lines

Quoted 56 lines

Quoted 5 lines

This is a hack...sorry, just joking ;-) Seriously, I think the situation with the userspace subdevs is a bit different. Because with one API we directly expose some functionality for applications, with other we code it in the kernel, to make the devices appear uniform at user space.

Not sure if I understood you. V4L2 export drivers functionality to userspace in an uniform way. MC api is for special applications that might need to access some internal functions on embedded devices.

Of course, there are some cases where it doesn't make sense to export a subdev control via V4L2 API.

Quoted 10 lines

Quoted 9 lines

Quoted 4 lines

Also, the sensor subdev can be configured in the video node driver as well as through the subdev device node. Both APIs can do the same thing but in order to let the subdev API work as expected the video node driver must be forbidden to configure the subdev.

Why? For the sensor, a V4L2 API call will look just like a bridge driver call. The subdev will need a mutex anyway, as two MC applications may be opening it simultaneously. I can't see why it should forbid changing the control from the bridge driver call.

Quoted 12 lines

Please do not forget there might be more than one subdev to configure and that the bridge itself is also a subdev (which exposes a scaler interface, for instance). A situation pretty much like in Figure 4.4
[1] (after the scaler there is also a video node to configure, but we may assume that pixel resolution at the scaler pad 1 is same as at the video node). Assuming the format and crop configuration flow is from sensor to host scaler direction, if we have tried to configure _all_ subdevs when the last stage of the pipeline is configured (i.e. video node) the whole scaler and crop/composition configuration we have been destroyed at that time. And there is more to configure than
VIDIOC_S_FMT can do.

Think from users perspective: all user wants is to see a video of a given resolution. S_FMT (and a few other VIDIOC_* calls) have everything that the user wants: the desired resolution, framerate and format.

Specialized applications indeed need more, in order to get the best images for certain types of usages. So, MC is there.

Such applications will probably need to know exactly what's the sensor, what are their bugs, how it is connected, what are the DSP blocks in the patch, how the DSP algorithms are implemented, etc, in order to obtain the the perfect image.

Even on embedded devices like smartphones and tablets, I predict that both types of applications will be developed and used: people may use a generic application like flash player, and an specialized application provided by the manufacturer. Users can even develop their own applications generic apps using V4L2 directly, at the devices that allow that.

Quoted 7 lines

Quoted 2 lines

As I said before: both application types are welcome. We just need to warrant that a pure V4L application will work reasonably well.

That's why we have libv4l. The driver simply doesn't receive enough information to configure the hardware correctly from the VIDIOC_* calls. And as mentioned above, 3A algorithms, required by "simple" V4L2 applications, need to be implemented in userspace anyway.

It is OK to improve users experience via libv4l. What I'm saying is that it is NOT OK to remove V4L2 API support from the driver, forcing users to use some hardware plugin at libv4l.

Quoted 4 lines

Let me be clear on this. I'm *NOT* advocating removing V4L2 API support from any driver (well, on the drivers I can currently think of, if you show me a wifi driver that implements a V4L2 interface I might change my mind :-)).

This thread is all about a patch series partially removing V4L2 API support.

Quoted 5 lines

The V4L2 API has been designed mostly for desktop hardware. Thanks to its clean design we can use it for embedded hardware, even though it requires extensions. What we need to do is to define which parts of the whole API apply as-is to embedded hardware, and which should better be left unused.

Agreed. I'm all in favor on discussing this topic. However, before he have such iscussions and come into a common understanding, I won't take any patch removing V4L2 API support, nor I'll accept any driver that won't allow it to be usable by a V4L2 userspace application.

Quoted 45 lines

Quoted 44 lines

Quoted 34 lines

Quoted 34 lines

Quoted 2 lines

Allowing the bridge driver to configure subdevs at all times would prevent the subdev/MC API to work.

Well, then we need to think on an alternative for that. It seems an interesting theme for the media workshop at the Kernel Summit/2011.

Quoted 13 lines

Quoted 13 lines

Quoted 2 lines

There is a conflict there that in order to use 'optional' API the 'main' API behaviour must be affected....

It is optional from userspace perspective. A V4L2-only application should be able to work with all drivers. However, a MC-aware application will likely be specific for some hardware, as it will need to know some device-specific stuff.

Both kinds of applications are welcome, but dropping support for V4L2-only applications is the wrong thing to do.

Quoted 1 line

And I really cant use V4L2 API only as is because it's too limited.

Quoted 7 lines

Quoted 1 line

Why?

For instance there is really yet no support for scaler and composition onto a target buffer in the Video Capture Interface (we also use sensors with built in scalers). It's difficult to efficiently manage capture/preview pipelines. It is impossible to discover the system topology.

Scaler were always supported by V4L2: if the resolution specified by S_FMT is not what the sensor provides, then scale. All non-embedded drivers with sensor or bridge supports scale does that.

Composition is not properly supported yet. It could make sense to add it to V4L. How do you think MC API would help with composite?

Quoted 7 lines

Quoted 4 lines

Managing capture/preview pipelines will require some support at V4L2 level. This is a problem that needs to be addressed there anyway, as buffers for preview/capture need to be allocated. There's an RFC about that, but I don't think it covers the pipelines for it.

Managing pipelines is a policy decision, and needs to be implemented in userspace. Once again, the solution here is libv4l.

If V4L2 API is not enough, implementing it on libv4l won't solve, as userspace apps will use V4L2 API for requresting it.

Quoted 3 lines

We need to let userspace configure the pipeline. That's what the MC + V4L2 APIs are for. The driver must no make policy decisions though, that must be left to userspace.

Showing only the hardware that is supported by an embedded device is not a policy decision. It is hardware detection or platform data configuration.

Hi Mauro,

Quoted 6 lines

Quoted 2 lines

My point is that the ISP driver developer can't know in advance which sensor will be used systems that don't exist yet.

As far as such hardware is projected, someone will know. It is a simple trivial patch to associate a new hardware with a hardware profile at the platform data.

Platform data must contain hardware descriptions only, not policies. This is even clearer now that ARM is moving away from board code to the Device Tree.

Quoted 6 lines

Also, on most cases, probing a sensor is as trivial as reading a sensor ID during device probe. This applies, for example, for all Omnivision sensors.

We do things like that all the times for PC world, as nobody knows what webcam someone would plug on his PC.

Sorry, but that's not related. You simply can't decide in an embedded ISP driver how to deal with sensor controls, as the system will be used in a too wide variety of applications and hardware configurations. All controls need to be exposed, period.

Quoted 7 lines

Quoted 4 lines

Quoted 2 lines

I never saw an embedded hardware that allows physically changing the sensor.

Beagleboard + pluggable sensor board.

Development systems like beagleboard, pandaboard, Exynos SMDK, etc, aren't embeeded hardware. They're development kits.

People create end-user products based on those kits. That make them first- class embedded hardware like any other.

Quoted 7 lines

I don't mind if, for those kits the developer that is playing with it has to pass a mode parameter and/or run some open harware-aware small application that makes the driver to select the sensor type he is using, but, if the hardware is, instead, a N9 or a Galaxy Tab (or whatever embedded hardware), the driver should expose just the sensors that exists on such hardware. It shouldn't be ever allowed to change it on userspace, using whatever API on those hardware.

Who talked about changing sensors from userspace on those systems ? Platform data (regardless of whether it comes from board code, device tree, or something else) will contain a hardware description, and the kernel will create the right devices and load the right drivers. The issue we're discussing is how to expose controls for those devices to userspace, and that needs to be done through subdev nodes.

Quoted 87 lines

Quoted 86 lines

Quoted 10 lines

Quoted 2 lines

Even if you did, fine image quality tuning requires accessing pretty much all controls individually anyway.

The same is also true for non-embedded hardware. The only situation where V4L2 API is not enough is when there are two controls of the same type active. For example, 2 active volume controls, one at the audio demod, and another at the bridge. There may have some cases where you can do the same thing at the sensor or at a DSP block. This is where MC API gives an improvement, by allowing changing both, instead of just one of the controls.

To be precise it's the V4L2 subdev userspace API that allows that, not the MC API.

Quoted 59 lines

Quoted 59 lines

Quoted 58 lines

Quoted 5 lines

This is a hack...sorry, just joking ;-) Seriously, I think the situation with the userspace subdevs is a bit different. Because with one API we directly expose some functionality for applications, with other we code it in the kernel, to make the devices appear uniform at user space.

Not sure if I understood you. V4L2 export drivers functionality to userspace in an uniform way. MC api is for special applications that might need to access some internal functions on embedded devices.

Of course, there are some cases where it doesn't make sense to export a subdev control via V4L2 API.

Quoted 10 lines

Quoted 9 lines

Quoted 4 lines

Also, the sensor subdev can be configured in the video node driver as well as through the subdev device node. Both APIs can do the same thing but in order to let the subdev API work as expected the video node driver must be forbidden to configure the subdev.

Why? For the sensor, a V4L2 API call will look just like a bridge driver call. The subdev will need a mutex anyway, as two MC applications may be opening it simultaneously. I can't see why it should forbid changing the control from the bridge driver call.

Quoted 12 lines

Please do not forget there might be more than one subdev to configure and that the bridge itself is also a subdev (which exposes a scaler interface, for instance). A situation pretty much like in Figure 4.4
[1] (after the scaler there is also a video node to configure, but we may assume that pixel resolution at the scaler pad 1 is same as at the video node). Assuming the format and crop configuration flow is from sensor to host scaler direction, if we have tried to configure _all_ subdevs when the last stage of the pipeline is configured (i.e. video node) the whole scaler and crop/composition configuration we have been destroyed at that time. And there is more to configure than
VIDIOC_S_FMT can do.

Think from users perspective: all user wants is to see a video of a given resolution. S_FMT (and a few other VIDIOC_* calls) have everything that the user wants: the desired resolution, framerate and format.

Specialized applications indeed need more, in order to get the best images for certain types of usages. So, MC is there.

Such applications will probably need to know exactly what's the sensor, what are their bugs, how it is connected, what are the DSP blocks in the patch, how the DSP algorithms are implemented, etc, in order to obtain the the perfect image.

Even on embedded devices like smartphones and tablets, I predict that both types of applications will be developed and used: people may use a generic application like flash player, and an specialized application provided by the manufacturer. Users can even develop their own applications generic apps using V4L2 directly, at the devices that allow that.

As I said before: both application types are welcome. We just need to warrant that a pure V4L application will work reasonably well.

Quoted 8 lines

Quoted 4 lines

That's why we have libv4l. The driver simply doesn't receive enough information to configure the hardware correctly from the VIDIOC_* calls. And as mentioned above, 3A algorithms, required by "simple" V4L2 applications, need to be implemented in userspace anyway.

It is OK to improve users experience via libv4l. What I'm saying is that it is NOT OK to remove V4L2 API support from the driver, forcing users to use some hardware plugin at libv4l.

Let me be clear on this. I'm *NOT* advocating removing V4L2 API support from any driver (well, on the drivers I can currently think of, if you show me a wifi driver that implements a V4L2 interface I might change my mind :-)).

Quoted 2 lines

This thread is all about a patch series partially removing V4L2 API support.

Because that specific part of the API doesn't make sense for this use case. You wouldn't object to removing S_INPUT support from a video output driver, as it wouldn't make sense either.

Quoted 7 lines

Quoted 6 lines

The V4L2 API has been designed mostly for desktop hardware. Thanks to its clean design we can use it for embedded hardware, even though it requires extensions. What we need to do is to define which parts of the whole API apply as-is to embedded hardware, and which should better be left unused.

Quoted 4 lines

Agreed. I'm all in favor on discussing this topic. However, before he have such iscussions and come into a common understanding, I won't take any patch removing V4L2 API support, nor I'll accept any driver that won't allow it to be usable by a V4L2 userspace application.

A pure V4L2 userspace application, knowing about video device nodes only, can still use the driver. Not all advanced features will be available.

Em 16-08-2011 01:57, Laurent Pinchart escreveu:

Quoted 10 lines

Quoted 6 lines

Quoted 2 lines

My point is that the ISP driver developer can't know in advance which sensor will be used systems that don't exist yet.

As far as such hardware is projected, someone will know. It is a simple trivial patch to associate a new hardware with a hardware profile at the platform data.

Platform data must contain hardware descriptions only, not policies. This is even clearer now that ARM is moving away from board code to the Device Tree.

Again, a cell phone with one frontal camera and one hear camera has two sensor inputs only. This is not a "policy". It is a hardware constraint. The driver should allow setting the pipeline for both sensors via S_INPUT, otherwise a V4L2 only userspace application won't work.

It is as simple as that.

Quoted 7 lines

Quoted 6 lines

Also, on most cases, probing a sensor is as trivial as reading a sensor ID during device probe. This applies, for example, for all Omnivision sensors.

We do things like that all the times for PC world, as nobody knows what webcam someone would plug on his PC.

Quoted 4 lines

Sorry, but that's not related. You simply can't decide in an embedded ISP driver how to deal with sensor controls, as the system will be used in a too wide variety of applications and hardware configurations. All controls need to be exposed, period.

We're not talking about controls. We're talking about providing the needed V4L2 support to allow an userspace application to access the hardware sensor.

Quoted 8 lines

Quoted 7 lines

Quoted 4 lines

Quoted 2 lines

I never saw an embedded hardware that allows physically changing the sensor.

Beagleboard + pluggable sensor board.

Development systems like beagleboard, pandaboard, Exynos SMDK, etc, aren't embeeded hardware. They're development kits.

Quoted 2 lines

People create end-user products based on those kits. That make them first- class embedded hardware like any other.

No doubt they should be supported, but it doesn't make sense to create tons of input pipelines to be used for S_INPUT for each different type of possible sensor. Somehow, userspace needs to tell what's the sensor that he attached to the hardware, or the driver should suport auto-detecting it.

In other words, I see 2 options for that:
1) add hardware auto-detection at the sensor logic. At driver probe,
try to probe all sensors, if it is a hardware development kit;

2) add one new parameter at the driver: "sensors". If the hardware
is one of those kits, this parameter will allow the developer to specify the used sensors. It is the same logic as we do with userspace TV and grabber cards without eeprom or any other way to auto-detect the hardware.

Quoted 8 lines

Quoted 7 lines

I don't mind if, for those kits the developer that is playing with it has to pass a mode parameter and/or run some open harware-aware small application that makes the driver to select the sensor type he is using, but, if the hardware is, instead, a N9 or a Galaxy Tab (or whatever embedded hardware), the driver should expose just the sensors that exists on such hardware. It shouldn't be ever allowed to change it on userspace, using whatever API on those hardware.

Quoted 6 lines

Who talked about changing sensors from userspace on those systems ? Platform data (regardless of whether it comes from board code, device tree, or something else) will contain a hardware description, and the kernel will create the right devices and load the right drivers. The issue we're discussing is how to expose controls for those devices to userspace, and that needs to be done through subdev nodes.

The issue that is under discussion is the removal of S_INPUT from the samsung driver, and the comments at the patches that talks about removing V4L2 API support in favor of using a MC-only API for some fundamental things.

For example, with a patch like this one, only one sensor will be supported without the MC API (either the front or back sensor on a multi-sensor camera):
http://git.infradead.org/users/kmpark/linux-2.6-samsung/commit/47751733a32
2a241927f9238b8ab1441389c9c41

Look at the comment on this patch also:

	http://git.infradead.org/users/kmpark/linux-2.6-samsung/commit/c6fb462c38b

e60a45d16a29a9e56c886ee0aa08c

What is called "API compatibility mode" is not clear, but it transmitted me that the idea is to expose the controls only via subnodes.

Those are the rationale for those discussions: V4L2 API is not being deprecated in favor of MC API, e. g. controls shouldn't be hidden from the V4L2 API without a good reason.

Quoted 92 lines

Quoted 91 lines

Quoted 80 lines

Quoted 10 lines

Quoted 2 lines

Even if you did, fine image quality tuning requires accessing pretty much all controls individually anyway.

The same is also true for non-embedded hardware. The only situation where V4L2 API is not enough is when there are two controls of the same type active. For example, 2 active volume controls, one at the audio demod, and another at the bridge. There may have some cases where you can do the same thing at the sensor or at a DSP block. This is where MC API gives an improvement, by allowing changing both, instead of just one of the controls.

To be precise it's the V4L2 subdev userspace API that allows that, not the MC API.

Quoted 66 lines

Quoted 65 lines

Quoted 58 lines

Quoted 5 lines

This is a hack...sorry, just joking ;-) Seriously, I think the situation with the userspace subdevs is a bit different. Because with one API we directly expose some functionality for applications, with other we code it in the kernel, to make the devices appear uniform at user space.

Not sure if I understood you. V4L2 export drivers functionality to userspace in an uniform way. MC api is for special applications that might need to access some internal functions on embedded devices.

Of course, there are some cases where it doesn't make sense to export a subdev control via V4L2 API.

Quoted 11 lines

Quoted 10 lines

Quoted 5 lines

Also, the sensor subdev can be configured in the video node driver as well as through the subdev device node. Both APIs can do the same thing but in order to let the subdev API work as expected the video node driver must be forbidden to configure the subdev.

Why? For the sensor, a V4L2 API call will look just like a bridge driver call. The subdev will need a mutex anyway, as two MC applications may be opening it simultaneously. I can't see why it should forbid changing the control from the bridge driver call.

Quoted 13 lines

Please do not forget there might be more than one subdev to configure and that the bridge itself is also a subdev (which exposes a scaler interface, for instance). A situation pretty much like in Figure 4.4 [1] (after the scaler there is also a video node to configure, but we may assume that pixel resolution at the scaler pad 1 is same as at the video node). Assuming the format and crop configuration flow is from sensor to host scaler direction, if we have tried to configure _all_ subdevs when the last stage of the pipeline is configured (i.e. video node) the whole scaler and crop/composition
configuration we have been destroyed at that time. And there is more to configure than
VIDIOC_S_FMT can do.

Think from users perspective: all user wants is to see a video of a given resolution. S_FMT (and a few other VIDIOC_* calls) have everything that the user wants: the desired resolution, framerate and format.

Specialized applications indeed need more, in order to get the best images for certain types of usages. So, MC is there.

Such applications will probably need to know exactly what's the sensor, what are their bugs, how it is connected, what are the DSP blocks in the patch, how the DSP algorithms are implemented, etc, in order to obtain the the perfect image.

Even on embedded devices like smartphones and tablets, I predict that both types of applications will be developed and used: people may use a generic application like flash player, and an specialized application provided by the manufacturer. Users can even develop their own applications generic apps using V4L2 directly, at the devices that allow that.

Quoted 2 lines

As I said before: both application types are welcome. We just need to warrant that a pure V4L application will work reasonably well.

That's why we have libv4l. The driver simply doesn't receive enough information to configure the hardware correctly from the VIDIOC_* calls. And as mentioned above, 3A algorithms, required by "simple" V4L2 applications, need to be implemented in userspace anyway.

Quoted 8 lines

Quoted 3 lines

It is OK to improve users experience via libv4l. What I'm saying is that it is NOT OK to remove V4L2 API support from the driver, forcing users to use some hardware plugin at libv4l.

Let me be clear on this. I'm *NOT* advocating removing V4L2 API support from any driver (well, on the drivers I can currently think of, if you show me a wifi driver that implements a V4L2 interface I might change my mind :-)).

This thread is all about a patch series partially removing V4L2 API support.

Quoted 3 lines

Because that specific part of the API doesn't make sense for this use case. You wouldn't object to removing S_INPUT support from a video output driver, as it wouldn't make sense either.

A device with two sensors input where just one node can be switched to use either input is a typical case where S_INPUT needs to be provided.

Quoted 6 lines

Quoted 6 lines

Quoted 5 lines

The V4L2 API has been designed mostly for desktop hardware. Thanks to its clean design we can use it for embedded hardware, even though it requires extensions. What we need to do is to define which parts of the whole API apply as-is to embedded hardware, and which should better be left unused.

Quoted 7 lines

Quoted 4 lines

Agreed. I'm all in favor on discussing this topic. However, before he have such iscussions and come into a common understanding, I won't take any patch removing V4L2 API support, nor I'll accept any driver that won't allow it to be usable by a V4L2 userspace application.

A pure V4L2 userspace application, knowing about video device nodes only, can still use the driver. Not all advanced features will be available.

That's exactly what I'm saying.

Hi Mauro,

Quoted 17 lines · On Tuesday 16 August 2011 17:30:47 Mauro Carvalho Chehab wrote:

Em 16-08-2011 01:57, Laurent Pinchart escreveu:

Quoted 10 lines

Quoted 6 lines

Quoted 2 lines

My point is that the ISP driver developer can't know in advance which sensor will be used systems that don't exist yet.

As far as such hardware is projected, someone will know. It is a simple trivial patch to associate a new hardware with a hardware profile at the platform data.

Platform data must contain hardware descriptions only, not policies. This is even clearer now that ARM is moving away from board code to the Device Tree.

Again, a cell phone with one frontal camera and one hear camera has two sensor inputs only. This is not a "policy". It is a hardware constraint. The driver should allow setting the pipeline for both sensors via S_INPUT, otherwise a V4L2 only userspace application won't work.

Quoted 1 line

It is as simple as that.

When capturing from the main sensor on the OMAP3 ISP you need to capture raw data to memory on a video node, feed it back to the hardware through another video node, and finally capture it on a third video node. A V4L2-only userspace application won't work. That's how the hardware is, we can't do much about that.

Quoted 8 lines

Quoted 8 lines

Quoted 7 lines

Also, on most cases, probing a sensor is as trivial as reading a sensor ID during device probe. This applies, for example, for all Omnivision sensors.

We do things like that all the times for PC world, as nobody knows what webcam someone would plug on his PC.

Quoted 8 lines

Quoted 4 lines

Sorry, but that's not related. You simply can't decide in an embedded ISP driver how to deal with sensor controls, as the system will be used in a too wide variety of applications and hardware configurations. All controls need to be exposed, period.

We're not talking about controls. We're talking about providing the needed V4L2 support to allow an userspace application to access the hardware sensor.

OK, so we're discussing S_INPUT. Let's discuss controls later :-)

Quoted 8 lines

Quoted 8 lines

Quoted 7 lines

Quoted 4 lines

Quoted 2 lines

I never saw an embedded hardware that allows physically changing the sensor.

Beagleboard + pluggable sensor board.

Development systems like beagleboard, pandaboard, Exynos SMDK, etc, aren't embeeded hardware. They're development kits.

Quoted 8 lines

Quoted 2 lines

People create end-user products based on those kits. That make them first- class embedded hardware like any other.

No doubt they should be supported, but it doesn't make sense to create tons of input pipelines to be used for S_INPUT for each different type of possible sensor. Somehow, userspace needs to tell what's the sensor that he attached to the hardware, or the driver should suport auto-detecting it.

We're not creating tons of input pipelines. Look at
http://www.ideasonboard.org/media/omap3isp.ps , every video node (in yellow) has its purpose.

Quoted 3 lines

In other words, I see 2 options for that:
1) add hardware auto-detection at the sensor logic. At driver probe,
try to probe all sensors, if it is a hardware development kit;

We've worked quite hard to remove I2C device probing from the kernel, let's not add it back.

Quoted 4 lines

2) add one new parameter at the driver: "sensors". If the hardware
is one of those kits, this parameter will allow the developer to specify the used sensors. It is the same logic as we do with userspace TV and grabber cards without eeprom or any other way to auto-detect the hardware.

This will likely be done through the Device Tree.

Quoted 28 lines

Quoted 8 lines

Quoted 7 lines

I don't mind if, for those kits the developer that is playing with it has to pass a mode parameter and/or run some open harware-aware small application that makes the driver to select the sensor type he is using, but, if the hardware is, instead, a N9 or a Galaxy Tab (or whatever embedded hardware), the driver should expose just the sensors that exists on such hardware. It shouldn't be ever allowed to change it on userspace, using whatever API on those hardware.

Quoted 6 lines

Who talked about changing sensors from userspace on those systems ? Platform data (regardless of whether it comes from board code, device tree, or something else) will contain a hardware description, and the kernel will create the right devices and load the right drivers. The issue we're discussing is how to expose controls for those devices to userspace, and that needs to be done through subdev nodes.

The issue that is under discussion is the removal of S_INPUT from the samsung driver, and the comments at the patches that talks about removing V4L2 API support in favor of using a MC-only API for some fundamental things.

For example, with a patch like this one, only one sensor will be supported without the MC API (either the front or back sensor on a multi-sensor camera):
http://git.infradead.org/users/kmpark/linux-2.6-samsung/commit/47751733a32
2a241927f9238b8ab1441389c9c41

Look at the comment on this patch also:

	http://git.infradead.org/users/kmpark/linux-2.6-samsung/commit/c6fb462c38b

Quoted 8 lines

e60a45d16a29a9e56c886ee0aa08c

What is called "API compatibility mode" is not clear, but it transmitted me that the idea is to expose the controls only via subnodes.

Those are the rationale for those discussions: V4L2 API is not being deprecated in favor of MC API, e. g. controls shouldn't be hidden from the V4L2 API without a good reason.

Controls need to move to subdev nodes for embedded devices because there's simply no way to expose multiple identical controls through a video node. Please also have a look at the diagram I linked to above, and tell me though which video node sensor controls should be exposed. There's no simple answer to that.

Quoted 93 lines

Quoted 93 lines

Quoted 92 lines

Quoted 81 lines

Quoted 11 lines

Quoted 3 lines

Even if you did, fine image quality tuning requires accessing pretty much all controls individually anyway.

The same is also true for non-embedded hardware. The only situation where V4L2 API is not enough is when there are two controls of the same type active. For example, 2 active volume controls, one at the audio demod, and another at the bridge. There may have some cases where you can do the same thing at the sensor or at a DSP block. This is where MC API gives an improvement, by allowing changing both, instead of just one of the controls.

To be precise it's the V4L2 subdev userspace API that allows that, not the MC API.

Quoted 66 lines

Quoted 65 lines

Quoted 58 lines

Quoted 5 lines

This is a hack...sorry, just joking ;-) Seriously, I think the situation with the userspace subdevs is a bit different. Because with one API we directly expose some functionality for applications, with other we code it in the kernel, to make the devices appear uniform at user space.

Not sure if I understood you. V4L2 export drivers functionality to userspace in an uniform way. MC api is for special applications that might need to access some internal functions on embedded devices.

Of course, there are some cases where it doesn't make sense to export a subdev control via V4L2 API.

Quoted 11 lines

Quoted 10 lines

Quoted 5 lines

Also, the sensor subdev can be configured in the video node driver as well as through the subdev device node. Both APIs can do the same thing but in order to let the subdev API work as expected the video node driver must be forbidden to configure the subdev.

Why? For the sensor, a V4L2 API call will look just like a bridge driver call. The subdev will need a mutex anyway, as two MC applications may be opening it simultaneously. I can't see why it should forbid changing the control from the bridge driver call.

Quoted 13 lines

Please do not forget there might be more than one subdev to configure and that the bridge itself is also a subdev (which exposes a scaler interface, for instance). A situation pretty much like in Figure 4.4 [1] (after the scaler there is also a video node to configure, but we may assume that pixel resolution at the scaler pad 1 is same as at the video node). Assuming the format and crop configuration flow is from sensor to host scaler direction, if we have tried to configure _all_ subdevs when the last stage of the pipeline is configured (i.e. video node) the whole scaler and crop/composition
configuration we have been destroyed at that time. And there is more to configure than
VIDIOC_S_FMT can do.

Think from users perspective: all user wants is to see a video of a given resolution. S_FMT (and a few other VIDIOC_* calls) have everything that the user wants: the desired resolution, framerate and format.

Specialized applications indeed need more, in order to get the best images for certain types of usages. So, MC is there.

Such applications will probably need to know exactly what's the sensor, what are their bugs, how it is connected, what are the DSP blocks in the patch, how the DSP algorithms are implemented, etc, in order to obtain the the perfect image.

Even on embedded devices like smartphones and tablets, I predict that both types of applications will be developed and used: people may use a generic application like flash player, and an specialized application provided by the manufacturer. Users can even develop their own applications generic apps using V4L2 directly, at the devices that allow that.

Quoted 2 lines

As I said before: both application types are welcome. We just need to warrant that a pure V4L application will work reasonably well.

That's why we have libv4l. The driver simply doesn't receive enough information to configure the hardware correctly from the VIDIOC_* calls. And as mentioned above, 3A algorithms, required by "simple" V4L2 applications, need to be implemented in userspace anyway.

Quoted 8 lines

Quoted 3 lines

It is OK to improve users experience via libv4l. What I'm saying is that it is NOT OK to remove V4L2 API support from the driver, forcing users to use some hardware plugin at libv4l.

Let me be clear on this. I'm *NOT* advocating removing V4L2 API support from any driver (well, on the drivers I can currently think of, if you show me a wifi driver that implements a V4L2 interface I might change my mind :-)).

This thread is all about a patch series partially removing V4L2 API support.

Quoted 6 lines

Quoted 3 lines

Because that specific part of the API doesn't make sense for this use case. You wouldn't object to removing S_INPUT support from a video output driver, as it wouldn't make sense either.

A device with two sensors input where just one node can be switched to use either input is a typical case where S_INPUT needs to be provided.

No. S_INPUT shouldn't be use to select between sensors. The hardware pipeline is more complex than just that. We can't make it all fit in the S_INPUT API.

For instance, when switching between a b&w and a color sensor you will need to reconfigure the whole pipeline to select the right gamma table, white balance parameters, color conversion matrix, ... That's not something we want to hardcode in the kernel. This needs to be done from userspace.

Em 16-08-2011 08:44, Laurent Pinchart escreveu:

Quoted 18 lines · On Tuesday 16 August 2011 17:30:47 Mauro Carvalho Chehab wrote:

Quoted 17 lines

Em 16-08-2011 01:57, Laurent Pinchart escreveu:

Quoted 10 lines

Quoted 6 lines

Quoted 2 lines

My point is that the ISP driver developer can't know in advance which sensor will be used systems that don't exist yet.

As far as such hardware is projected, someone will know. It is a simple trivial patch to associate a new hardware with a hardware profile at the platform data.

Platform data must contain hardware descriptions only, not policies. This is even clearer now that ARM is moving away from board code to the Device Tree.

Again, a cell phone with one frontal camera and one hear camera has two sensor inputs only. This is not a "policy". It is a hardware constraint. The driver should allow setting the pipeline for both sensors via S_INPUT, otherwise a V4L2 only userspace application won't work.

Quoted 7 lines

Quoted 1 line

It is as simple as that.

When capturing from the main sensor on the OMAP3 ISP you need to capture raw data to memory on a video node, feed it back to the hardware through another video node, and finally capture it on a third video node. A V4L2-only userspace application won't work. That's how the hardware is, we can't do much about that.

The raw data conversion is one of the functions that libv4l should do. So, if you've already submitted the patches for libv4l to do the hardware loopback trick, or add a function there to convert the raw data into a common format, that should be ok. Otherwise, we have a problem, that needs to be fixed.

Quoted 35 lines

Quoted 7 lines

Quoted 7 lines

Quoted 6 lines

Also, on most cases, probing a sensor is as trivial as reading a sensor ID during device probe. This applies, for example, for all Omnivision sensors.

We do things like that all the times for PC world, as nobody knows what webcam someone would plug on his PC.

Quoted 8 lines

Quoted 4 lines

Sorry, but that's not related. You simply can't decide in an embedded ISP driver how to deal with sensor controls, as the system will be used in a too wide variety of applications and hardware configurations. All controls need to be exposed, period.

We're not talking about controls. We're talking about providing the needed V4L2 support to allow an userspace application to access the hardware sensor.

OK, so we're discussing S_INPUT. Let's discuss controls later :-)

Quoted 16 lines

Quoted 10 lines

Quoted 7 lines

Quoted 4 lines

Quoted 2 lines

I never saw an embedded hardware that allows physically changing the sensor.

Beagleboard + pluggable sensor board.

Development systems like beagleboard, pandaboard, Exynos SMDK, etc, aren't embeeded hardware. They're development kits.

People create end-user products based on those kits. That make them first- class embedded hardware like any other.

No doubt they should be supported, but it doesn't make sense to create tons of input pipelines to be used for S_INPUT for each different type of possible sensor. Somehow, userspace needs to tell what's the sensor that he attached to the hardware, or the driver should suport auto-detecting it.

Quoted 3 lines

We're not creating tons of input pipelines. Look at
http://www.ideasonboard.org/media/omap3isp.ps , every video node (in yellow) has its purpose.

Not sure if I it understood well. The subdevs 8-11 are the sensors, right?

Quoted 5 lines

Quoted 4 lines

In other words, I see 2 options for that:
1) add hardware auto-detection at the sensor logic. At driver probe,

try to probe all sensors, if it is a hardware development kit;

Quoted 2 lines

We've worked quite hard to remove I2C device probing from the kernel, let's not add it back.

We do I2C probing on several drivers. It is there for devices where the cards entry is not enough to identify the hardware. For example, two different devices with the same USB ID generally uses that. If the hardware information is not enough, there's nothing wrong on doing that.

Quoted 7 lines

Quoted 6 lines

2) add one new parameter at the driver: "sensors". If the hardware

is one of those kits, this parameter will allow the developer to specify the used sensors. It is the same logic as we do with userspace TV and grabber cards without eeprom or any other way to auto-detect the hardware.

Quoted 1 line

This will likely be done through the Device Tree.

I don't mind much about the way it is implemented, but it should be there on a place where people can find it and use it.

Devices that requires the user to pass some parameters to the Kernel in order for the driver to find the hardware are economic class devices. A first class device should work as-is, after the driver is loaded.

Quoted 39 lines

Quoted 30 lines

Quoted 8 lines

Quoted 7 lines

I don't mind if, for those kits the developer that is playing with it has to pass a mode parameter and/or run some open harware-aware small application that makes the driver to select the sensor type he is using, but, if the hardware is, instead, a N9 or a Galaxy Tab (or whatever embedded hardware), the driver should expose just the sensors that exists on such hardware. It shouldn't be ever allowed to change it on userspace, using whatever API on those hardware.

Quoted 6 lines

Who talked about changing sensors from userspace on those systems ? Platform data (regardless of whether it comes from board code, device tree, or something else) will contain a hardware description, and the kernel will create the right devices and load the right drivers. The issue we're discussing is how to expose controls for those devices to userspace, and that needs to be done through subdev nodes.

The issue that is under discussion is the removal of S_INPUT from the samsung driver, and the comments at the patches that talks about removing V4L2 API support in favor of using a MC-only API for some fundamental things.

For example, with a patch like this one, only one sensor will be supported without the MC API (either the front or back sensor on a multi-sensor camera):
http://git.infradead.org/users/kmpark/linux-2.6-samsung/commit/47751733a
32 2a241927f9238b8ab1441389c9c41

Look at the comment on this patch also:

	http://git.infradead.org/users/kmpark/linux-2.6-samsung/commit/c6fb462c
	38b

Quoted 8 lines

e60a45d16a29a9e56c886ee0aa08c

What is called "API compatibility mode" is not clear, but it transmitted me that the idea is to expose the controls only via subnodes.

Those are the rationale for those discussions: V4L2 API is not being deprecated in favor of MC API, e. g. controls shouldn't be hidden from the V4L2 API without a good reason.

Quoted 5 lines

Controls need to move to subdev nodes for embedded devices because there's simply no way to expose multiple identical controls through a video node. Please also have a look at the diagram I linked to above, and tell me though which video node sensor controls should be exposed. There's no simple answer to that.

Again, not sure if I understood well your diagram. What device will be controlling and receiving the video streaming from the sensor? /dev/video0?

If so, this is the one that should be exposing the controls for the selected input sensor.

Quoted 105 lines

Quoted 94 lines

Quoted 94 lines

Quoted 93 lines

Quoted 82 lines

Quoted 10 lines

Quoted 2 lines

Even if you did, fine image quality tuning requires accessing pretty much all controls individually anyway.

The same is also true for non-embedded hardware. The only situation where V4L2 API is not enough is when there are two controls of the same type active. For example, 2 active volume controls, one at the audio demod, and another at the bridge. There may have some cases where you can do the same thing at the sensor or at a DSP block. This is where MC API gives an improvement, by allowing changing both, instead of just one of the controls.

To be precise it's the V4L2 subdev userspace API that allows that, not the MC API.

Quoted 68 lines

Quoted 67 lines

Quoted 60 lines

Quoted 5 lines

This is a hack...sorry, just joking ;-) Seriously, I think the situation with the userspace subdevs is a bit different. Because with one API we directly expose some functionality for applications, with other we code it in the kernel, to make the devices appear uniform at user space.

Not sure if I understood you. V4L2 export drivers functionality to userspace in an uniform way. MC api is for special applications that might need to access some internal functions on embedded devices.

Of course, there are some cases where it doesn't make sense to export a subdev control via V4L2 API.

Quoted 12 lines

Quoted 11 lines

Quoted 5 lines

Also, the sensor subdev can be configured in the video node driver as well as through the subdev device node. Both APIs can do the same thing but in order to let the subdev API work as expected the video node driver must be forbidden to configure the subdev.

Why? For the sensor, a V4L2 API call will look just like a bridge driver call. The subdev will need a mutex anyway, as two MC applications may be opening it simultaneously. I can't see why it should forbid changing the control from the bridge driver call.

Quoted 13 lines

Please do not forget there might be more than one subdev to configure and that the bridge itself is also a subdev (which exposes a scaler interface, for instance). A situation pretty much like in Figure 4.4 [1] (after the scaler there is also a video node to configure, but we may assume that pixel resolution at the scaler pad 1 is same as at the video node). Assuming the format and crop configuration flow is from sensor to host scaler direction, if we have tried to configure _all_ subdevs when the last stage of the pipeline is configured (i.e. video node) the whole scaler and crop/composition
configuration we have been destroyed at that time. And there is more to configure than
VIDIOC_S_FMT can do.

Think from users perspective: all user wants is to see a video of a given resolution. S_FMT (and a few other VIDIOC_* calls) have everything that the user wants: the desired resolution, framerate and format.

Specialized applications indeed need more, in order to get the best images for certain types of usages. So, MC is there.

Such applications will probably need to know exactly what's the sensor, what are their bugs, how it is connected, what are the DSP blocks in the patch, how the DSP algorithms are implemented, etc, in order to obtain the the perfect image.

Even on embedded devices like smartphones and tablets, I predict that both types of applications will be developed and used: people may use a generic application like flash player, and an specialized application provided by the manufacturer. Users can even develop their own applications generic apps using V4L2 directly, at the devices that allow that.

Quoted 2 lines

As I said before: both application types are welcome. We just need to warrant that a pure V4L application will work reasonably well.

That's why we have libv4l. The driver simply doesn't receive enough information to configure the hardware correctly from the VIDIOC_* calls. And as mentioned above, 3A algorithms, required by "simple" V4L2 applications, need to be implemented in userspace anyway.

Quoted 8 lines

Quoted 3 lines

It is OK to improve users experience via libv4l. What I'm saying is that it is NOT OK to remove V4L2 API support from the driver, forcing users to use some hardware plugin at libv4l.

Let me be clear on this. I'm *NOT* advocating removing V4L2 API support from any driver (well, on the drivers I can currently think of, if you show me a wifi driver that implements a V4L2 interface I might change my mind :-)).

This thread is all about a patch series partially removing V4L2 API support.

Quoted 6 lines

Quoted 3 lines

Because that specific part of the API doesn't make sense for this use case. You wouldn't object to removing S_INPUT support from a video output driver, as it wouldn't make sense either.

A device with two sensors input where just one node can be switched to use either input is a typical case where S_INPUT needs to be provided.

No. S_INPUT shouldn't be use to select between sensors. The hardware pipeline is more complex than just that. We can't make it all fit in the S_INPUT API.

Quoted 5 lines

For instance, when switching between a b&w and a color sensor you will need to reconfigure the whole pipeline to select the right gamma table, white balance parameters, color conversion matrix, ... That's not something we want to hardcode in the kernel. This needs to be done from userspace.

This is something that, if it is not written somehwere, no userspace applications not developed by the hardware vendor will ever work.

I don't see any code for that any at the kernel or at libv4l. Am I missing something?

Hi Mauro,

Quoted 27 lines · On Wednesday 17 August 2011 00:36:44 Mauro Carvalho Chehab wrote:

Em 16-08-2011 08:44, Laurent Pinchart escreveu:

Quoted 18 lines · On Tuesday 16 August 2011 17:30:47 Mauro Carvalho Chehab wrote:

Quoted 17 lines

Em 16-08-2011 01:57, Laurent Pinchart escreveu:

Quoted 10 lines

Quoted 6 lines

Quoted 2 lines

My point is that the ISP driver developer can't know in advance which sensor will be used systems that don't exist yet.

As far as such hardware is projected, someone will know. It is a simple trivial patch to associate a new hardware with a hardware profile at the platform data.

Platform data must contain hardware descriptions only, not policies. This is even clearer now that ARM is moving away from board code to the Device Tree.

Again, a cell phone with one frontal camera and one hear camera has two sensor inputs only. This is not a "policy". It is a hardware constraint. The driver should allow setting the pipeline for both sensors via S_INPUT, otherwise a V4L2 only userspace application won't work.

Quoted 7 lines

Quoted 1 line

It is as simple as that.

When capturing from the main sensor on the OMAP3 ISP you need to capture raw data to memory on a video node, feed it back to the hardware through another video node, and finally capture it on a third video node. A V4L2-only userspace application won't work. That's how the hardware is, we can't do much about that.

Quoted 5 lines

The raw data conversion is one of the functions that libv4l should do. So, if you've already submitted the patches for libv4l to do the hardware loopback trick, or add a function there to convert the raw data into a common format, that should be ok. Otherwise, we have a problem, that needs to be fixed.

That's not what this is about. Bayer to YUV conversion needs to happen in hardware in this case for performance reasons. The hardware will also perform scaling on YUV, as well as many image processing tasks (white balance, defect pixel correction, gamma correction, noise filtering, ...).

Quoted 35 lines

Quoted 35 lines

Quoted 7 lines

Quoted 7 lines

Quoted 6 lines

Also, on most cases, probing a sensor is as trivial as reading a sensor ID during device probe. This applies, for example, for all Omnivision sensors.

We do things like that all the times for PC world, as nobody knows what webcam someone would plug on his PC.

Quoted 8 lines

Quoted 4 lines

Sorry, but that's not related. You simply can't decide in an embedded ISP driver how to deal with sensor controls, as the system will be used in a too wide variety of applications and hardware configurations. All controls need to be exposed, period.

We're not talking about controls. We're talking about providing the needed V4L2 support to allow an userspace application to access the hardware sensor.

OK, so we're discussing S_INPUT. Let's discuss controls later :-)

Quoted 16 lines

Quoted 10 lines

Quoted 7 lines

Quoted 4 lines

Quoted 2 lines

I never saw an embedded hardware that allows physically changing the sensor.

Beagleboard + pluggable sensor board.

Development systems like beagleboard, pandaboard, Exynos SMDK, etc, aren't embeeded hardware. They're development kits.

People create end-user products based on those kits. That make them first- class embedded hardware like any other.

No doubt they should be supported, but it doesn't make sense to create tons of input pipelines to be used for S_INPUT for each different type of possible sensor. Somehow, userspace needs to tell what's the sensor that he attached to the hardware, or the driver should suport auto-detecting it.

Quoted 5 lines

Quoted 3 lines

We're not creating tons of input pipelines. Look at
http://www.ideasonboard.org/media/omap3isp.ps , every video node (in yellow) has its purpose.

Not sure if I it understood well. The subdevs 8-11 are the sensors, right?

et8ek8 and vs6555 are the sensors. ad5820 is the lens controller and adp1653 the flash controller. All other subdevs (green blocks) are part of the ISP.

Quoted 5 lines

Quoted 5 lines

Quoted 4 lines

In other words, I see 2 options for that:
1) add hardware auto-detection at the sensor logic. At driver probe,

try to probe all sensors, if it is a hardware development kit;

Quoted 8 lines

Quoted 2 lines

We've worked quite hard to remove I2C device probing from the kernel, let's not add it back.

We do I2C probing on several drivers. It is there for devices where the cards entry is not enough to identify the hardware. For example, two different devices with the same USB ID generally uses that. If the hardware information is not enough, there's nothing wrong on doing that.

Except that probing might destroy the hardware in the general case. We can only probe I2C devices on a bus that we know will not contain any other sensitive devices.

Quoted 7 lines

Quoted 7 lines

Quoted 6 lines

2) add one new parameter at the driver: "sensors". If the hardware

is one of those kits, this parameter will allow the developer to specify the used sensors. It is the same logic as we do with userspace TV and grabber cards without eeprom or any other way to auto-detect the hardware.

Quoted 8 lines

Quoted 1 line

This will likely be done through the Device Tree.

I don't mind much about the way it is implemented, but it should be there on a place where people can find it and use it.

Devices that requires the user to pass some parameters to the Kernel in order for the driver to find the hardware are economic class devices. A first class device should work as-is, after the driver is loaded.

I don't think there has ever been any disagreement on this, we can consider the matter settled.

Quoted 45 lines

Quoted 39 lines

Quoted 30 lines

Quoted 8 lines

Quoted 7 lines

I don't mind if, for those kits the developer that is playing with it has to pass a mode parameter and/or run some open harware-aware small application that makes the driver to select the sensor type he is using, but, if the hardware is, instead, a N9 or a Galaxy Tab (or whatever embedded hardware), the driver should expose just the sensors that exists on such hardware. It shouldn't be ever allowed to change it on userspace, using whatever API on those hardware.

Quoted 6 lines

Who talked about changing sensors from userspace on those systems ? Platform data (regardless of whether it comes from board code, device tree, or something else) will contain a hardware description, and the kernel will create the right devices and load the right drivers. The issue we're discussing is how to expose controls for those devices to userspace, and that needs to be done through subdev nodes.

The issue that is under discussion is the removal of S_INPUT from the samsung driver, and the comments at the patches that talks about removing V4L2 API support in favor of using a MC-only API for some fundamental things.

For example, with a patch like this one, only one sensor will be supported without the MC API (either the front or back sensor on a multi-sensor camera):
http://git.infradead.org/users/kmpark/linux-2.6-samsung/commit/47751733a
32 2a241927f9238b8ab1441389c9c41

Look at the comment on this patch also:

	http://git.infradead.org/users/kmpark/linux-2.6-samsung/commit/c6fb462c
	38b

Quoted 8 lines

e60a45d16a29a9e56c886ee0aa08c

What is called "API compatibility mode" is not clear, but it transmitted me that the idea is to expose the controls only via subnodes.

Those are the rationale for those discussions: V4L2 API is not being deprecated in favor of MC API, e. g. controls shouldn't be hidden from the V4L2 API without a good reason.

Quoted 5 lines

Controls need to move to subdev nodes for embedded devices because there's simply no way to expose multiple identical controls through a video node. Please also have a look at the diagram I linked to above, and tell me though which video node sensor controls should be exposed. There's no simple answer to that.

Quoted 5 lines

Again, not sure if I understood well your diagram. What device will be controlling and receiving the video streaming from the sensor? /dev/video0?

If so, this is the one that should be exposing the controls for the selected input sensor.

When capturing data from the main sensor (et8ek8), a common pipeline is to capture data from /dev/video1, feed it back through /dev/video0 and capture the final result from /dev/video6. Another common alternative is to capture it from /dev/video2, feed it back to /dev/video3 and capture the final result from /dev/video6. Regardless of which above configuration is used, applications can also need to capture images on /dev/video4 and /dev/video6 concurrently. This all depends on what kind of image the application wants and what hardware processing it wants to apply.

This isn't even a complex case, there are much more complex hardware out there that we want to support.

Quoted 111 lines

Quoted 110 lines

Quoted 94 lines

Quoted 94 lines

Quoted 93 lines

Quoted 82 lines

Quoted 10 lines

Quoted 2 lines

Even if you did, fine image quality tuning requires accessing pretty much all controls individually anyway.

The same is also true for non-embedded hardware. The only situation where V4L2 API is not enough is when there are two controls of the same type active. For example, 2 active volume controls, one at the audio demod, and another at the bridge. There may have some cases where you can do the same thing at the sensor or at a DSP block. This is where MC API gives an improvement, by allowing changing both, instead of just one of the controls.

To be precise it's the V4L2 subdev userspace API that allows that, not the MC API.

Quoted 68 lines

Quoted 67 lines

Quoted 60 lines

Quoted 5 lines

This is a hack...sorry, just joking ;-) Seriously, I think the situation with the userspace subdevs is a bit different. Because with one API we directly expose some functionality for applications, with other we code it in the kernel, to make the devices appear uniform at user space.

Not sure if I understood you. V4L2 export drivers functionality to userspace in an uniform way. MC api is for special applications that might need to access some internal functions on embedded devices.

Of course, there are some cases where it doesn't make sense to export a subdev control via V4L2 API.

Quoted 12 lines

Quoted 11 lines

Quoted 5 lines

Also, the sensor subdev can be configured in the video node
driver as well as through the subdev device node. Both APIs can
do the same thing but in order to let the subdev API work as
expected the video node driver must be forbidden to configure
the subdev.

Why? For the sensor, a V4L2 API call will look just like a bridge driver call. The subdev will need a mutex anyway, as two MC applications may be opening it simultaneously. I can't see why it should forbid changing the control from the bridge driver call.

Quoted 13 lines

Please do not forget there might be more than one subdev to configure and that the bridge itself is also a subdev (which exposes a scaler interface, for instance). A situation pretty much like in Figure 4.4 [1] (after the scaler there is also a video node to configure, but we may assume that pixel resolution at the scaler pad 1 is same as at the video node). Assuming the format and crop configuration flow is from sensor to host scaler direction, if we have tried to configure _all_ subdevs when the last stage of the pipeline is configured (i.e. video node) the whole scaler and crop/composition
configuration we have been destroyed at that time. And there is more to configure than
VIDIOC_S_FMT can do.

Think from users perspective: all user wants is to see a video of a given resolution. S_FMT (and a few other VIDIOC_* calls) have everything that the user wants: the desired resolution, framerate and format.

Specialized applications indeed need more, in order to get the best images for certain types of usages. So, MC is there.

Such applications will probably need to know exactly what's the sensor, what are their bugs, how it is connected, what are the DSP blocks in the patch, how the DSP algorithms are implemented, etc, in order to obtain the the perfect image.

Even on embedded devices like smartphones and tablets, I predict that both types of applications will be developed and used: people may use a generic application like flash player, and an specialized application provided by the manufacturer. Users can even develop their own applications generic apps using V4L2 directly, at the devices that allow that.

Quoted 2 lines

As I said before: both application types are welcome. We just need to warrant that a pure V4L application will work reasonably well.

That's why we have libv4l. The driver simply doesn't receive enough information to configure the hardware correctly from the VIDIOC_* calls. And as mentioned above, 3A algorithms, required by "simple" V4L2 applications, need to be implemented in userspace anyway.

Quoted 8 lines

Quoted 3 lines

It is OK to improve users experience via libv4l. What I'm saying is that it is NOT OK to remove V4L2 API support from the driver, forcing users to use some hardware plugin at libv4l.

Let me be clear on this. I'm *NOT* advocating removing V4L2 API support from any driver (well, on the drivers I can currently think of, if you show me a wifi driver that implements a V4L2 interface I might change my mind :-)).

This thread is all about a patch series partially removing V4L2 API support.

Quoted 6 lines

Quoted 3 lines

Because that specific part of the API doesn't make sense for this use case. You wouldn't object to removing S_INPUT support from a video output driver, as it wouldn't make sense either.

A device with two sensors input where just one node can be switched to use either input is a typical case where S_INPUT needs to be provided.

No. S_INPUT shouldn't be use to select between sensors. The hardware pipeline is more complex than just that. We can't make it all fit in the S_INPUT API.

For instance, when switching between a b&w and a color sensor you will need to reconfigure the whole pipeline to select the right gamma table, white balance parameters, color conversion matrix, ... That's not something we want to hardcode in the kernel. This needs to be done from userspace.

Quoted 5 lines

This is something that, if it is not written somehwere, no userspace applications not developed by the hardware vendor will ever work.

I don't see any code for that any at the kernel or at libv4l. Am I missing something?

Code for that needs to be written in libv4l. It's not there yet as I don't think we have any hardware for this particular example at the moment :-)

Em 17-08-2011 00:57, Laurent Pinchart escreveu:

Quoted 299 lines

Hi Mauro,

Quoted 27 lines · On Wednesday 17 August 2011 00:36:44 Mauro Carvalho Chehab wrote:

Em 16-08-2011 08:44, Laurent Pinchart escreveu:

Quoted 18 lines · On Tuesday 16 August 2011 17:30:47 Mauro Carvalho Chehab wrote:

Quoted 17 lines

Em 16-08-2011 01:57, Laurent Pinchart escreveu:

Quoted 10 lines

Quoted 6 lines

Quoted 2 lines

My point is that the ISP driver developer can't know in advance which sensor will be used systems that don't exist yet.

As far as such hardware is projected, someone will know. It is a simple trivial patch to associate a new hardware with a hardware profile at the platform data.

Platform data must contain hardware descriptions only, not policies. This is even clearer now that ARM is moving away from board code to the Device Tree.

Again, a cell phone with one frontal camera and one hear camera has two sensor inputs only. This is not a "policy". It is a hardware constraint. The driver should allow setting the pipeline for both sensors via S_INPUT, otherwise a V4L2 only userspace application won't work.

Quoted 7 lines

Quoted 1 line

It is as simple as that.

When capturing from the main sensor on the OMAP3 ISP you need to capture raw data to memory on a video node, feed it back to the hardware through another video node, and finally capture it on a third video node. A V4L2-only userspace application won't work. That's how the hardware is, we can't do much about that.

Quoted 5 lines

The raw data conversion is one of the functions that libv4l should do. So, if you've already submitted the patches for libv4l to do the hardware loopback trick, or add a function there to convert the raw data into a common format, that should be ok. Otherwise, we have a problem, that needs to be fixed.

That's not what this is about. Bayer to YUV conversion needs to happen in hardware in this case for performance reasons. The hardware will also perform scaling on YUV, as well as many image processing tasks (white balance, defect pixel correction, gamma correction, noise filtering, ...).

Quoted 35 lines

Quoted 35 lines

Quoted 7 lines

Quoted 7 lines

Quoted 6 lines

Also, on most cases, probing a sensor is as trivial as reading a sensor ID during device probe. This applies, for example, for all Omnivision sensors.

We do things like that all the times for PC world, as nobody knows what webcam someone would plug on his PC.

Quoted 8 lines

Quoted 4 lines

Sorry, but that's not related. You simply can't decide in an embedded ISP driver how to deal with sensor controls, as the system will be used in a too wide variety of applications and hardware configurations. All controls need to be exposed, period.

We're not talking about controls. We're talking about providing the needed V4L2 support to allow an userspace application to access the hardware sensor.

OK, so we're discussing S_INPUT. Let's discuss controls later :-)

Quoted 16 lines

Quoted 10 lines

Quoted 7 lines

Quoted 4 lines

Quoted 2 lines

I never saw an embedded hardware that allows physically changing the sensor.

Beagleboard + pluggable sensor board.

Development systems like beagleboard, pandaboard, Exynos SMDK, etc, aren't embeeded hardware. They're development kits.

People create end-user products based on those kits. That make them first- class embedded hardware like any other.

No doubt they should be supported, but it doesn't make sense to create tons of input pipelines to be used for S_INPUT for each different type of possible sensor. Somehow, userspace needs to tell what's the sensor that he attached to the hardware, or the driver should suport auto-detecting it.

Quoted 5 lines

Quoted 3 lines

We're not creating tons of input pipelines. Look at
http://www.ideasonboard.org/media/omap3isp.ps , every video node (in yellow) has its purpose.

Not sure if I it understood well. The subdevs 8-11 are the sensors, right?

et8ek8 and vs6555 are the sensors. ad5820 is the lens controller and adp1653 the flash controller. All other subdevs (green blocks) are part of the ISP.

Quoted 5 lines

Quoted 5 lines

Quoted 4 lines

In other words, I see 2 options for that:
1) add hardware auto-detection at the sensor logic. At driver probe,

try to probe all sensors, if it is a hardware development kit;

Quoted 8 lines

Quoted 2 lines

We've worked quite hard to remove I2C device probing from the kernel, let's not add it back.

We do I2C probing on several drivers. It is there for devices where the cards entry is not enough to identify the hardware. For example, two different devices with the same USB ID generally uses that. If the hardware information is not enough, there's nothing wrong on doing that.

Except that probing might destroy the hardware in the general case. We can only probe I2C devices on a bus that we know will not contain any other sensitive devices.

Quoted 7 lines

Quoted 7 lines

Quoted 6 lines

2) add one new parameter at the driver: "sensors". If the hardware

is one of those kits, this parameter will allow the developer to specify the used sensors. It is the same logic as we do with userspace TV and grabber cards without eeprom or any other way to auto-detect the hardware.

Quoted 8 lines

Quoted 1 line

This will likely be done through the Device Tree.

I don't mind much about the way it is implemented, but it should be there on a place where people can find it and use it.

Devices that requires the user to pass some parameters to the Kernel in order for the driver to find the hardware are economic class devices. A first class device should work as-is, after the driver is loaded.

I don't think there has ever been any disagreement on this, we can consider the matter settled.

Quoted 45 lines

Quoted 39 lines

Quoted 30 lines

Quoted 8 lines

Quoted 7 lines

I don't mind if, for those kits the developer that is playing with it has to pass a mode parameter and/or run some open harware-aware small application that makes the driver to select the sensor type he is using, but, if the hardware is, instead, a N9 or a Galaxy Tab (or whatever embedded hardware), the driver should expose just the sensors that exists on such hardware. It shouldn't be ever allowed to change it on userspace, using whatever API on those hardware.

Quoted 6 lines

Who talked about changing sensors from userspace on those systems ? Platform data (regardless of whether it comes from board code, device tree, or something else) will contain a hardware description, and the kernel will create the right devices and load the right drivers. The issue we're discussing is how to expose controls for those devices to userspace, and that needs to be done through subdev nodes.

The issue that is under discussion is the removal of S_INPUT from the samsung driver, and the comments at the patches that talks about removing V4L2 API support in favor of using a MC-only API for some fundamental things.

For example, with a patch like this one, only one sensor will be supported without the MC API (either the front or back sensor on a multi-sensor camera):
http://git.infradead.org/users/kmpark/linux-2.6-samsung/commit/47751733a
32 2a241927f9238b8ab1441389c9c41

Look at the comment on this patch also:

	http://git.infradead.org/users/kmpark/linux-2.6-samsung/commit/c6fb462c
	38b

Quoted 8 lines

e60a45d16a29a9e56c886ee0aa08c

What is called "API compatibility mode" is not clear, but it transmitted me that the idea is to expose the controls only via subnodes.

Those are the rationale for those discussions: V4L2 API is not being deprecated in favor of MC API, e. g. controls shouldn't be hidden from the V4L2 API without a good reason.

Quoted 5 lines

Controls need to move to subdev nodes for embedded devices because there's simply no way to expose multiple identical controls through a video node. Please also have a look at the diagram I linked to above, and tell me though which video node sensor controls should be exposed. There's no simple answer to that.

Quoted 5 lines

Again, not sure if I understood well your diagram. What device will be controlling and receiving the video streaming from the sensor? /dev/video0?

If so, this is the one that should be exposing the controls for the selected input sensor.

When capturing data from the main sensor (et8ek8), a common pipeline is to capture data from /dev/video1, feed it back through /dev/video0 and capture the final result from /dev/video6. Another common alternative is to capture it from /dev/video2, feed it back to /dev/video3 and capture the final result from /dev/video6. Regardless of which above configuration is used, applications can also need to capture images on /dev/video4 and /dev/video6 concurrently. This all depends on what kind of image the application wants and what hardware processing it wants to apply.

This isn't even a complex case, there are much more complex hardware out there that we want to support.

Quoted 114 lines

Quoted 105 lines

Quoted 94 lines

Quoted 94 lines

Quoted 93 lines

Quoted 82 lines

Quoted 10 lines

Quoted 2 lines

Even if you did, fine image quality tuning requires accessing pretty much all controls individually anyway.

The same is also true for non-embedded hardware. The only situation where V4L2 API is not enough is when there are two controls of the same type active. For example, 2 active volume controls, one at the audio demod, and another at the bridge. There may have some cases where you can do the same thing at the sensor or at a DSP block. This is where MC API gives an improvement, by allowing changing both, instead of just one of the controls.

To be precise it's the V4L2 subdev userspace API that allows that, not the MC API.

Quoted 68 lines

Quoted 67 lines

Quoted 60 lines

Quoted 5 lines

This is a hack...sorry, just joking ;-) Seriously, I think the situation with the userspace subdevs is a bit different. Because with one API we directly expose some functionality for applications, with other we code it in the kernel, to make the devices appear uniform at user space.

Not sure if I understood you. V4L2 export drivers functionality to userspace in an uniform way. MC api is for special applications that might need to access some internal functions on embedded devices.

Of course, there are some cases where it doesn't make sense to export a subdev control via V4L2 API.

Quoted 12 lines

Quoted 11 lines

Also, the sensor subdev can be configured in the video node
driver as well as through the subdev device node. Both APIs can
do the same thing but in order to let the subdev API work as
expected the video node driver must be forbidden to configure
the subdev.

Why? For the sensor, a V4L2 API call will look just like a
bridge driver call. The subdev will need a mutex anyway, as two
MC applications may be opening it simultaneously. I can't see
why it should forbid changing the control from the bridge
driver call.

Quoted 13 lines

Please do not forget there might be more than one subdev to configure and that the bridge itself is also a subdev (which exposes a scaler interface, for instance). A situation pretty much like in Figure 4.4 [1] (after the scaler there is also a video node to configure, but we may assume that pixel resolution at the scaler pad 1 is same as at the video node). Assuming the format and crop configuration flow is from sensor to host scaler direction, if we have tried to configure _all_ subdevs when the last stage of the pipeline is configured (i.e. video node) the whole scaler and crop/composition
configuration we have been destroyed at that time. And there is more to configure than
VIDIOC_S_FMT can do.

Think from users perspective: all user wants is to see a video of a given resolution. S_FMT (and a few other VIDIOC_* calls) have everything that the user wants: the desired resolution, framerate and format.

Specialized applications indeed need more, in order to get the best images for certain types of usages. So, MC is there.

Such applications will probably need to know exactly what's the sensor, what are their bugs, how it is connected, what are the DSP blocks in the patch, how the DSP algorithms are implemented, etc, in order to obtain the the perfect image.

Even on embedded devices like smartphones and tablets, I predict that both types of applications will be developed and used: people may use a generic application like flash player, and an specialized application provided by the manufacturer. Users can even develop their own applications generic apps using V4L2 directly, at the devices that allow that.

Quoted 2 lines

As I said before: both application types are welcome. We just need to warrant that a pure V4L application will work reasonably well.

That's why we have libv4l. The driver simply doesn't receive enough information to configure the hardware correctly from the VIDIOC_* calls. And as mentioned above, 3A algorithms, required by "simple" V4L2 applications, need to be implemented in userspace anyway.

Quoted 8 lines

Quoted 3 lines

It is OK to improve users experience via libv4l. What I'm saying is that it is NOT OK to remove V4L2 API support from the driver, forcing users to use some hardware plugin at libv4l.

Let me be clear on this. I'm *NOT* advocating removing V4L2 API support from any driver (well, on the drivers I can currently think of, if you show me a wifi driver that implements a V4L2 interface I might change my mind :-)).

This thread is all about a patch series partially removing V4L2 API support.

Quoted 6 lines

Quoted 3 lines

Because that specific part of the API doesn't make sense for this use case. You wouldn't object to removing S_INPUT support from a video output driver, as it wouldn't make sense either.

A device with two sensors input where just one node can be switched to use either input is a typical case where S_INPUT needs to be provided.

No. S_INPUT shouldn't be use to select between sensors. The hardware pipeline is more complex than just that. We can't make it all fit in the S_INPUT API.

Quoted 5 lines

For instance, when switching between a b&w and a color sensor you will need to reconfigure the whole pipeline to select the right gamma table, white balance parameters, color conversion matrix, ... That's not something we want to hardcode in the kernel. This needs to be done from userspace.

This is something that, if it is not written somehwere, no userspace applications not developed by the hardware vendor will ever work.

Quoted 6 lines

Quoted 2 lines

I don't see any code for that any at the kernel or at libv4l. Am I missing something?

Code for that needs to be written in libv4l. It's not there yet as I don't think we have any hardware for this particular example at the moment :-)

As no pure V4L2 application would set the pipelines as you've said, and no libv4l code exists yet,

that means that either:
1) pure V4L application support is broken;
2) it is possible to select between the main sensor and the
secondary sensor via S_INPUT on /dev/video1, and to control the sensor via this node, in order to set bright, contrast, apperture time and exposition. Performance will be sacrificed, as the bayer->YUV conversion and 3A algorithms will be done unaccelerated in libv4l. The picture will also not look fine, as no noise reduction, etc will be done, but support for a pure V4L application requirement is satisfied.

Note also that even a generic MC-aware application will not work, as there's nothing at the MC pipeline information that shows how to proper configure the pipelines to get the expected result. Such application needs to have an internal database that associates each pipeline seen via the MC API to the processor type, and have the pipeline policy configurations coded internally.

Regards,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi everybody,

Quoted 20 lines · Mauro Carvalho Chehab

Em 17-08-2011 00:57, Laurent Pinchart escreveu:

Quoted 19 lines

Hi Mauro,

Quoted 16 lines · On Wednesday 17 August 2011 00:36:44 Mauro Carvalho Chehab wrote:

Em 16-08-2011 08:44, Laurent Pinchart escreveu:

Quoted 11 lines · On Tuesday 16 August 2011 17:30:47 Mauro Carvalho Chehab wrote:

Quoted 1 line

Em 16-08-2011 01:57, Laurent Pinchart escreveu:

No. S_INPUT shouldn't be use to select between sensors. The hardware pipeline is more complex than just that. We can't make it all fit in the S_INPUT API.

For instance, when switching between a b&w and a color sensor you will need to reconfigure the whole pipeline to select the right gamma table, white balance parameters, color conversion matrix, ... That's not something we want to hardcode in the kernel. This needs to be done from userspace.

This is something that, if it is not written somehwere, no userspace applications not developed by the hardware vendor will ever work.

Quoted 8 lines

Quoted 6 lines

Quoted 2 lines

I don't see any code for that any at the kernel or at libv4l. Am I missing something?

Code for that needs to be written in libv4l. It's not there yet as I don't think we have any hardware for this particular example at the moment :-)

As no pure V4L2 application would set the pipelines as you've said, and no libv4l code exists yet,

Actually there is such code for OMAP3 ISP driver. Plug-in support in libv4l have been extended a little bit [1] and plugin-in which handle request for "regular" video device nodes /dev/video0 and /dev/video1 and translate them to MC and sub-device API have been posted here [2], but is still not merged.

Regards,
iivanov

[1] http://www.spinics.net/lists/linux-media/msg35570.html
[2] http://www.spinics.net/lists/linux-media/msg32539.html

Quoted 4 lines

For instance, when switching between a b&w and a color sensor you will need to reconfigure the whole pipeline to select the right gamma table, white balance parameters, color conversion matrix, ... That's not something we want to hardcode in the kernel. This needs to be done from userspace.

This is something that, if it is not written somehwere, no userspace applications not developed by the hardware vendor will ever work.

I don't see any code for that any at the kernel or at libv4l. Am I missing something?

Em 16-08-2011 01:57, Laurent Pinchart escreveu:

Quoted 8 lines

Hi Mauro,

Quoted 6 lines

Quoted 2 lines

My point is that the ISP driver developer can't know in advance which sensor will be used systems that don't exist yet.

As far as such hardware is projected, someone will know. It is a simple trivial patch to associate a new hardware with a hardware profile at the platform data.

Quoted 3 lines

Platform data must contain hardware descriptions only, not policies. This is even clearer now that ARM is moving away from board code to the Device Tree.

Again, a cell phone with one frontal camera and one hear camera has two sensor inputs only. This is not a "policy". It is a hardware constraint.

The driver should allow setting the pipeline for both sensors via S_INPUT, otherwise a V4L2 only userspace application won't work.

It is as simple as that.

Quoted 7 lines

Quoted 6 lines

Also, on most cases, probing a sensor is as trivial as reading a sensor ID during device probe. This applies, for example, for all Omnivision sensors.

We do things like that all the times for PC world, as nobody knows what webcam someone would plug on his PC.

Quoted 4 lines

Sorry, but that's not related. You simply can't decide in an embedded ISP driver how to deal with sensor controls, as the system will be used in a too wide variety of applications and hardware configurations. All controls need to be exposed, period.

We're not talking about controls. We're talking about providing the needed V4L2 support to allow an userspace application to access the hardware sensor.

Quoted 11 lines

Quoted 8 lines

Quoted 5 lines

Quoted 3 lines

I never saw an embedded hardware that allows physically changing the sensor.

Beagleboard + pluggable sensor board.

Development systems like beagleboard, pandaboard, Exynos SMDK, etc, aren't embeeded hardware. They're development kits.

People create end-user products based on those kits. That make them first- class embedded hardware like any other.

No doubt they should be supported, but it doesn't make sense to create tons of input pipelines to be used for S_INPUT for each different type of possible sensor. Somehow, userspace needs to tell what's the sensor that he attached to the hardware, or the driver should suport auto-detecting it.

In other words, I see 2 options for that:
1) add hardware auto-detection at the sensor logic. At driver probe,
try to probe all sensors, if it is a hardware development kit;

2) add one new parameter at the driver: "sensors". If the hardware
is one of those kits, this parameter will allow the developer to specify the used sensors. It is the same logic as we do with userspace TV and grabber cards without eeprom or any other way to auto-detect the hardware.

Quoted 9 lines

Quoted 8 lines

I don't mind if, for those kits the developer that is playing with it has to pass a mode parameter and/or run some open harware-aware small application that makes the driver to select the sensor type he is using, but, if the hardware is, instead, a N9 or a Galaxy Tab (or whatever embedded hardware), the driver should expose just the sensors that exists on such hardware. It shouldn't be ever allowed to change it on userspace, using whatever API on those hardware.

Quoted 6 lines

Who talked about changing sensors from userspace on those systems ? Platform data (regardless of whether it comes from board code, device tree, or something else) will contain a hardware description, and the kernel will create the right devices and load the right drivers. The issue we're discussing is how to expose controls for those devices to userspace, and that needs to be done through subdev nodes.

The issue that is under discussion is the removal of S_INPUT from the samsung driver, and the comments at the patches that talks about removing V4L2 API support in favor of using a MC-only API for some fundamental things.

For example, with a patch like this one, only one sensor will be supported without the MC API (either the front or back sensor on a multi-sensor camera):

	http://git.infradead.org/users/kmpark/linux-2.6-samsung/commit/47751733a322a241927f9238b8ab1441389c9c41

Look at the comment on this patch also:

	http://git.infradead.org/users/kmpark/linux-2.6-samsung/commit/c6fb462c38be60a45d16a29a9e56c886ee0aa08c

What is called "API compatibility mode" is not clear, but it transmitted me that the idea is to expose the controls only via subnodes.

Those are the rationale for those discussions: V4L2 API is not being deprecated in favor of MC API, e. g. controls shouldn't be hidden from the V4L2 API without a good reason.

Quoted 90 lines

Quoted 89 lines

Quoted 78 lines

Quoted 10 lines

Quoted 2 lines

Even if you did, fine image quality tuning requires accessing pretty much all controls individually anyway.

The same is also true for non-embedded hardware. The only situation where V4L2 API is not enough is when there are two controls of the same type active. For example, 2 active volume controls, one at the audio demod, and another at the bridge. There may have some cases where you can do the same thing at the sensor or at a DSP block. This is where MC API gives an improvement, by allowing changing both, instead of just one of the controls.

To be precise it's the V4L2 subdev userspace API that allows that, not the MC API.

Quoted 64 lines

Quoted 63 lines

Quoted 56 lines

Quoted 5 lines

This is a hack...sorry, just joking ;-) Seriously, I think the situation with the userspace subdevs is a bit different. Because with one API we directly expose some functionality for applications, with other we code it in the kernel, to make the devices appear uniform at user space.

Not sure if I understood you. V4L2 export drivers functionality to userspace in an uniform way. MC api is for special applications that might need to access some internal functions on embedded devices.

Of course, there are some cases where it doesn't make sense to export a subdev control via V4L2 API.

Quoted 10 lines

Quoted 9 lines

Quoted 4 lines

Also, the sensor subdev can be configured in the video node driver as well as through the subdev device node. Both APIs can do the same thing but in order to let the subdev API work as expected the video node driver must be forbidden to configure the subdev.

Why? For the sensor, a V4L2 API call will look just like a bridge driver call. The subdev will need a mutex anyway, as two MC applications may be opening it simultaneously. I can't see why it should forbid changing the control from the bridge driver call.

Quoted 12 lines

Please do not forget there might be more than one subdev to configure and that the bridge itself is also a subdev (which exposes a scaler interface, for instance). A situation pretty much like in Figure 4.4
[1] (after the scaler there is also a video node to configure, but we may assume that pixel resolution at the scaler pad 1 is same as at the video node). Assuming the format and crop configuration flow is from sensor to host scaler direction, if we have tried to configure _all_ subdevs when the last stage of the pipeline is configured (i.e. video node) the whole scaler and crop/composition configuration we have been destroyed at that time. And there is more to configure than
VIDIOC_S_FMT can do.

Think from users perspective: all user wants is to see a video of a given resolution. S_FMT (and a few other VIDIOC_* calls) have everything that the user wants: the desired resolution, framerate and format.

Specialized applications indeed need more, in order to get the best images for certain types of usages. So, MC is there.

Such applications will probably need to know exactly what's the sensor, what are their bugs, how it is connected, what are the DSP blocks in the patch, how the DSP algorithms are implemented, etc, in order to obtain the the perfect image.

Even on embedded devices like smartphones and tablets, I predict that both types of applications will be developed and used: people may use a generic application like flash player, and an specialized application provided by the manufacturer. Users can even develop their own applications generic apps using V4L2 directly, at the devices that allow that.

Quoted 2 lines

As I said before: both application types are welcome. We just need to warrant that a pure V4L application will work reasonably well.

That's why we have libv4l. The driver simply doesn't receive enough information to configure the hardware correctly from the VIDIOC_* calls. And as mentioned above, 3A algorithms, required by "simple" V4L2 applications, need to be implemented in userspace anyway.

Quoted 8 lines

Quoted 3 lines

It is OK to improve users experience via libv4l. What I'm saying is that it is NOT OK to remove V4L2 API support from the driver, forcing users to use some hardware plugin at libv4l.

Let me be clear on this. I'm *NOT* advocating removing V4L2 API support from any driver (well, on the drivers I can currently think of, if you show me a wifi driver that implements a V4L2 interface I might change my mind :-)).

This thread is all about a patch series partially removing V4L2 API support.

Quoted 3 lines

Because that specific part of the API doesn't make sense for this use case. You wouldn't object to removing S_INPUT support from a video output driver, as it wouldn't make sense either.

A device with two sensors input where just one node can be switched to use either input is a typical case where S_INPUT needs to be provided.

It seems that there are too many miss understandings or maybe we're just talking the same thing on different ways.

So, instead of answering again, let's re-start this discussion on a different way.

One of the requirements that it was discussed a lot on both mailing lists and on the Media Controllers meetings that we had (or, at least in the ones where I've participated) is that:

"A pure V4L2 userspace application, knowing about video device
nodes only, can still use the driver. Not all advanced features will be available."

This is easily said than done. Also, different understandings can be obtained by a simple phrase like that.

Hi Mauro,

Quoted 10 lines · Mauro Carvalho Chehab

It seems that there are too many miss understandings or maybe we're just talking the same thing on different ways.

So, instead of answering again, let's re-start this discussion on a different way.

One of the requirements that it was discussed a lot on both mailing lists and on the Media Controllers meetings that we had (or, at least in the ones where I've participated) is that:

Quoted 3 lines

"A pure V4L2 userspace application, knowing about video device
nodes only, can still use the driver. Not all advanced features will be available."

What does a term "a pure V4L2 userspace application" mean here ?

Does it also account an application which is linked to libv4l2 and uses calls specific to a particular hardware which are included there?

Em 20-08-2011 04:27, Sylwester Nawrocki escreveu:

Quoted 13 lines

Hi Mauro,

Quoted 10 lines · Mauro Carvalho Chehab

It seems that there are too many miss understandings or maybe we're just talking the same thing on different ways.

So, instead of answering again, let's re-start this discussion on a different way.

One of the requirements that it was discussed a lot on both mailing lists and on the Media Controllers meetings that we had (or, at least in the ones where I've participated) is that:

Quoted 5 lines

Quoted 3 lines

"A pure V4L2 userspace application, knowing about video device
nodes only, can still use the driver. Not all advanced features will be available."

What does a term "a pure V4L2 userspace application" mean here ?

The above quotation are exactly the Laurent's words that I took from one of his replies.

Quoted 2 lines

Does it also account an application which is linked to libv4l2 and uses calls specific to a particular hardware which are included there?

That's a good question. We need to properly define what it means, to avoid having libv4l abuses.

In other words, it seems ok to use libv4l to set pipelines via the MC API at open(), but it isn't ok to have an open() binary only libv4l plugin that will hook open and do the complete device initialization on userspace (I remember that one vendor once proposed a driver like that).

Also, from my side, I'd like to see both libv4l and kernel drivers being submitted together, if the new driver depends on a special libv4l support for it to work.

Hi Mauro,

Quoted 18 lines · Mauro Carvalho Chehab

Em 20-08-2011 04:27, Sylwester Nawrocki escreveu:

Quoted 17 lines

Hi Mauro,

Quoted 13 lines · Mauro Carvalho Chehab

It seems that there are too many miss understandings or maybe we're just talking the same thing on different ways.

So, instead of answering again, let's re-start this discussion on a different way.

One of the requirements that it was discussed a lot on both mailing lists and on the Media Controllers meetings that we had (or, at least in the ones where I've participated) is that:

"A pure V4L2 userspace application, knowing about video device
nodes only, can still use the driver. Not all advanced features will be available."

Quoted 4 lines

Quoted 1 line

What does a term "a pure V4L2 userspace application" mean here ?

The above quotation are exactly the Laurent's words that I took from one of his replies.

I would define this as an application which uses V4L2 but does not use Media controller or the V4L2 subdev interfaces nor is aware of any particular hardware device.

Quoted 6 lines

Quoted 2 lines

Does it also account an application which is linked to libv4l2 and uses calls specific to a particular hardware which are included there?

That's a good question. We need to properly define what it means, to avoid having libv4l abuses.

Quoted 8 lines

In other words, it seems ok to use libv4l to set pipelines via the MC API at open(), but it isn't ok to have an open() binary only libv4l plugin that will hook open and do the complete device initialization on userspace (I remember that one vendor once proposed a driver like that).

Also, from my side, I'd like to see both libv4l and kernel drivers being submitted together, if the new driver depends on a special libv4l support for it to work.

I agree with the above.

I do favour using libv4l to do the pipeline setup using MC and V4L2 subdev interfaces. This has the benefit that the driver provides just one interface to access different aspects of it, be it pipeline setup (Media controller), a control to change exposure time (V4L2 subdev) or queueing video buffer (V4L2). This means more simple and more maintainable drivers and less bugs in general.

Apart from what the drivers already provide on video nodea, to support a general purpose V4L2 application, libv4l plugin can do is (not exhaustive list):

- Perform pipeline setup using MC interface, possibly based on input selected using S_INPUT so that e.g. multiple sensors can be supported and
- implement {S,G,TRY}_EXT_CTRLS and QUERYCTRL using V4L2 subdev nodes as backend.

As the Media controller and V4L2 interfaces are standardised, I see no reason why this plugin could not be fully generic: only the configuration is device specific.

This configuration could be stored into a configuration file which is selected based on the system type. On embedded systems (ARMs at least, but anyway the vast majority is based on ARM) the board type is easily available for the user space applications in /proc/cpuinfo --- this example is from the Nokia N900:

---
Processor : ARMv7 Processor rev 3 (v7l)
BogoMIPS : 249.96
Features : swp half fastmult vfp edsp neon vfpv3
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x1
CPU part : 0xc08
CPU revision : 3

Hardware : Nokia RX-51 board
Revision : 2101
Serial : 0000000000000000
---

I think this would be a first step to support general purpose application on embedded systems.

The question I still have on this is that how should the user know which video node to access on an embedded system with a camera: the OMAP 3 ISP, for example, contains some eight video nodes which have different ISP blocks connected to them. Likely two of these nodes are useful for a general purpose application based on which image format it requests. It would make sense to provide generic applications information only on those devices they may meaningfully use.

Later on, more functionality could be added to better support hardware which supports e.g. different pixel formats, image sizes, scaling and crop. I'm not entirely certain if all of this is fully generic --- but I think the vast majority of it is --- at least converting from v4l2_mbus_framefmt pixelcode to v4l2_format.fmt is often quite hardware specific which must be taken into account by the generic plugin.

At that point many policy decisions must be made on how to use the hardware the best way, and that must be also present in the configuration file.

But perhaps I'm going too far with this now; we don't yet have a generic plugin providing basic functionality. We have the OMAP 3 ISP libv4l plugin which might be a good staring point for this work.

Em 24-08-2011 19:29, Sakari Ailus escreveu:

Quoted 7 lines

The question I still have on this is that how should the user know which video node to access on an embedded system with a camera: the OMAP 3 ISP, for example, contains some eight video nodes which have different ISP blocks connected to them. Likely two of these nodes are useful for a general purpose application based on which image format it requests. It would make sense to provide generic applications information only on those devices they may meaningfully use.

IMO, we should create a namespace device mapping for video devices. What I mean is that we should keep the "raw" V4L2 devices as:

	/dev/video??
But also recommend the creation of a new userspace map, like:
	/dev/webcam??
	/dev/tv??
	...

with is an alias for the actual device.

Something similar to dvd/cdrom aliases that already happen on most distros:

lrwxrwxrwx   1 root root           3 Ago 24 12:14 cdrom -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 cdrw -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 dvd -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 dvdrw -> sr0

Hi Mauro,

Quoted 1 line · On Thursday 25 August 2011 14:43:56 Mauro Carvalho Chehab wrote:

Em 24-08-2011 19:29, Sakari Ailus escreveu:

[snip]

Quoted 15 lines

Quoted 7 lines

The question I still have on this is that how should the user know which video node to access on an embedded system with a camera: the OMAP 3 ISP, for example, contains some eight video nodes which have different ISP blocks connected to them. Likely two of these nodes are useful for a general purpose application based on which image format it requests. It would make sense to provide generic applications information only on those devices they may meaningfully use.

IMO, we should create a namespace device mapping for video devices. What I mean is that we should keep the "raw" V4L2 devices as:

	/dev/video??
But also recommend the creation of a new userspace map, like:
	/dev/webcam??
	/dev/tv??
	...

Quoted 8 lines

with is an alias for the actual device.

Something similar to dvd/cdrom aliases that already happen on most distros:

lrwxrwxrwx   1 root root           3 Ago 24 12:14 cdrom -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 cdrw -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 dvd -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 dvdrw -> sr0

I've been toying with a similar idea. libv4l currently wraps /dev/video* device nodes and assumes a 1:1 relationship between a video device node and a video device. Should this assumption be somehow removed, replaced by a video device concept that wouldn't be tied to a single video device node ?

Quoted 31 lines

Hi Mauro,

Quoted 1 line · On Thursday 25 August 2011 14:43:56 Mauro Carvalho Chehab wrote:

Em 24-08-2011 19:29, Sakari Ailus escreveu:

[snip]

Quoted 15 lines

Quoted 7 lines

The question I still have on this is that how should the user know which video node to access on an embedded system with a camera: the OMAP 3 ISP, for example, contains some eight video nodes which have different ISP blocks connected to them. Likely two of these nodes are useful for a general purpose application based on which image format it requests. It would make sense to provide generic applications information only on those devices they may meaningfully use.

IMO, we should create a namespace device mapping for video devices. What I mean is that we should keep the "raw" V4L2 devices as:

	/dev/video??
But also recommend the creation of a new userspace map, like:
	/dev/webcam??
	/dev/tv??
	...

Quoted 8 lines

with is an alias for the actual device.

Something similar to dvd/cdrom aliases that already happen on most distros:

lrwxrwxrwx   1 root root           3 Ago 24 12:14 cdrom -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 cdrw -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 dvd -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 dvdrw -> sr0

Quoted 4 lines

I've been toying with a similar idea. libv4l currently wraps /dev/video* device nodes and assumes a 1:1 relationship between a video device node and a video device. Should this assumption be somehow removed, replaced by a video device concept that wouldn't be tied to a single video device node ?

Just as background information: the original idea was always that all v4l drivers would have a MC and that libv4l would use the information contained there as a helper (such as deciding which nodes would be the 'default' nodes for generic applications).

Since there is only one MC device node for each piece of video hardware that would make it much easier to discover what hardware there is and what video nodes to use.

I always liked that idea, although I know Mauro is opposed to having a MC for all v4l drivers.

While I am not opposed to creating such userspace maps I also think it is a bit of a poor-man's solution.

In particular I am worried that we get a
lot of those mappings (just think of ivtv with its 8 or 9 devices).

I can think of: webcam, tv, compressed (mpeg), tv-out, compressed-out, mem2mem.

But a 'tv' node might also be able to handle compressed video (depending on how the hardware is organized), so how do you handle that?

It can all
be solved, I'm sure, but I'm not sure if such userspace mappings will scale that well with the increasing hardware complexity.

Em 26-08-2011 11:16, Hans Verkuil escreveu:

Quoted 37 lines · On Friday, August 26, 2011 15:45:30 Laurent Pinchart wrote:

Quoted 31 lines

Hi Mauro,

Quoted 1 line · On Thursday 25 August 2011 14:43:56 Mauro Carvalho Chehab wrote:

Em 24-08-2011 19:29, Sakari Ailus escreveu:

[snip]

Quoted 15 lines

Quoted 7 lines

The question I still have on this is that how should the user know which video node to access on an embedded system with a camera: the OMAP 3 ISP, for example, contains some eight video nodes which have different ISP blocks connected to them. Likely two of these nodes are useful for a general purpose application based on which image format it requests. It would make sense to provide generic applications information only on those devices they may meaningfully use.

IMO, we should create a namespace device mapping for video devices. What I mean is that we should keep the "raw" V4L2 devices as:

	/dev/video??
But also recommend the creation of a new userspace map, like:
	/dev/webcam??
	/dev/tv??
	...

Quoted 8 lines

with is an alias for the actual device.

Something similar to dvd/cdrom aliases that already happen on most distros:

lrwxrwxrwx   1 root root           3 Ago 24 12:14 cdrom -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 cdrw -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 dvd -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 dvdrw -> sr0

Quoted 4 lines

I've been toying with a similar idea. libv4l currently wraps /dev/video* device nodes and assumes a 1:1 relationship between a video device node and a video device. Should this assumption be somehow removed, replaced by a video device concept that wouldn't be tied to a single video device node ?

Quoted 4 lines

Just as background information: the original idea was always that all v4l drivers would have a MC and that libv4l would use the information contained there as a helper (such as deciding which nodes would be the 'default' nodes for generic applications).

This is something that libv4l won't do: it is up to the userspace application to choose the device node to open. Ok, libv4l can have helper APIs for that, like the one I wrote, but even adding MC support on it may not solve the issues.

Quoted 4 lines

Since there is only one MC device node for each piece of video hardware that would make it much easier to discover what hardware there is and what video nodes to use.

Quoted 2 lines

I always liked that idea, although I know Mauro is opposed to having a MC for all v4l drivers.

It doesn't make sense to add MC for all V4L drivers. Not all devices are like ivtv with lots of device drivers. In a matter of fact, most supported devices create just one video node. Adding MC support for those devices will just increase the drivers complexity without _any_ reason, as those devices are fully configurable using the existing ioctl's.

Also, as I said before, and implemented at xawtv and at a v4l-utils library, the code may use sysfs for simpler devices. It shouldn't be hard to implement a mc aware code there, although I don't think that MC API is useful to discover what nodes are meant to be used for TV, encoder, decoder, webcams, etc. The only type information it currently provides is:

#define MEDIA_ENT_T_DEVNODE_V4L (MEDIA_ENT_T_DEVNODE + 1)
#define MEDIA_ENT_T_DEVNODE_FB (MEDIA_ENT_T_DEVNODE + 2)
#define MEDIA_ENT_T_DEVNODE_ALSA (MEDIA_ENT_T_DEVNODE + 3)
#define MEDIA_ENT_T_DEVNODE_DVB (MEDIA_ENT_T_DEVNODE + 4)

So, a MC aware application also needs to be a hardware-dependent application, as it will need to use something else, like the media entity name, to discover for what purpose a media node is meant to be used.

Quoted 2 lines

While I am not opposed to creating such userspace maps I also think it is a bit of a poor-man's solution.

The creation of per-type devices is part of the current API: radio and vbi nodes are examples of that (except that they aren't aliases, but real devices, but the idea is the same: different names for different types of usage).

Quoted 7 lines

In particular I am worried that we get a
lot of those mappings (just think of ivtv with its 8 or 9 devices).

I can think of: webcam, tv, compressed (mpeg), tv-out, compressed-out, mem2mem.

But a 'tv' node might also be able to handle compressed video (depending on how the hardware is organized), so how do you handle that?

Well, What you've called as "compressed" is, in IMO, "encoder". It probably makes sense to have, also "decoder".

I'm in doubt about "webcam", as there are some
grabber devices with analog camera inputs for video surveillance. Maybe "camera" is a better name for it.

Quoted 4 lines

It can all
be solved, I'm sure, but I'm not sure if such userspace mappings will scale that well with the increasing hardware complexity.

Not all video nodes would need an alias. Just the ones where it makes sense for an application to open it.

Quoted 55 lines

Em 26-08-2011 11:16, Hans Verkuil escreveu:

Quoted 37 lines · On Friday, August 26, 2011 15:45:30 Laurent Pinchart wrote:

Quoted 31 lines

Hi Mauro,

Quoted 1 line · On Thursday 25 August 2011 14:43:56 Mauro Carvalho Chehab wrote:

Em 24-08-2011 19:29, Sakari Ailus escreveu:

[snip]

Quoted 15 lines

Quoted 7 lines

The question I still have on this is that how should the user know which video node to access on an embedded system with a camera: the OMAP 3 ISP, for example, contains some eight video nodes which have different ISP blocks connected to them. Likely two of these nodes are useful for a general purpose application based on which image format it requests. It would make sense to provide generic applications information only on those devices they may meaningfully use.

IMO, we should create a namespace device mapping for video devices. What I mean is that we should keep the "raw" V4L2 devices as:

	/dev/video??
But also recommend the creation of a new userspace map, like:
	/dev/webcam??
	/dev/tv??
	...

Quoted 8 lines

with is an alias for the actual device.

Something similar to dvd/cdrom aliases that already happen on most distros:

lrwxrwxrwx   1 root root           3 Ago 24 12:14 cdrom -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 cdrw -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 dvd -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 dvdrw -> sr0

Quoted 4 lines

I've been toying with a similar idea. libv4l currently wraps /dev/video* device nodes and assumes a 1:1 relationship between a video device node and a video device. Should this assumption be somehow removed, replaced by a video device concept that wouldn't be tied to a single video device node ?

Quoted 4 lines

Just as background information: the original idea was always that all v4l drivers would have a MC and that libv4l would use the information contained there as a helper (such as deciding which nodes would be the 'default' nodes for generic applications).

This is something that libv4l won't do: it is up to the userspace application to choose the device node to open. Ok, libv4l can have helper APIs for that, like the one I wrote, but even adding MC support on it may not solve the issues.

Quoted 6 lines

Since there is only one MC device node for each piece of video hardware that would make it much easier to discover what hardware there is and what video nodes to use.

I always liked that idea, although I know Mauro is opposed to having a MC for all v4l drivers.

Quoted 5 lines

It doesn't make sense to add MC for all V4L drivers. Not all devices are like ivtv with lots of device drivers. In a matter of fact, most supported devices create just one video node. Adding MC support for those devices will just increase the drivers complexity without _any_ reason, as those devices are fully configurable using the existing ioctl's.

It's for consistency so applications know what to expect.

For all the simple
drivers you'd just need some simple core support to add a MC. What I always thought would be handy is for applications to just iterate over all MCs and show which video/dvb/audio hardware the user has in its system.

Quoted 7 lines

Also, as I said before, and implemented at xawtv and at a v4l-utils library, the code may use sysfs for simpler devices. It shouldn't be hard to implement a mc aware code there, although I don't think that MC API is useful to discover what nodes are meant to be used for TV, encoder, decoder, webcams, etc. The only type information it currently provides is:

Quoted 4 lines

#define MEDIA_ENT_T_DEVNODE_V4L (MEDIA_ENT_T_DEVNODE + 1)
#define MEDIA_ENT_T_DEVNODE_FB (MEDIA_ENT_T_DEVNODE + 2)
#define MEDIA_ENT_T_DEVNODE_ALSA (MEDIA_ENT_T_DEVNODE + 3)
#define MEDIA_ENT_T_DEVNODE_DVB (MEDIA_ENT_T_DEVNODE + 4)

That's because we never added meta information like that. As long as the MC is only used for SoC/complex drivers there is no point in adding such info.

It would be trivial to add precisely this type of information, though.

Quoted 11 lines

So, a MC aware application also needs to be a hardware-dependent application, as it will need to use something else, like the media entity name, to discover for what purpose a media node is meant to be used.

Quoted 2 lines

While I am not opposed to creating such userspace maps I also think it is a bit of a poor-man's solution.

The creation of per-type devices is part of the current API: radio and vbi nodes are examples of that (except that they aren't aliases, but real devices, but the idea is the same: different names for different types of usage).

That's why I'm not opposed to it. I'm just not sure how detailed/extensive that mapping should be.

Quoted 10 lines

Quoted 7 lines

In particular I am worried that we get a
lot of those mappings (just think of ivtv with its 8 or 9 devices).

I can think of: webcam, tv, compressed (mpeg), tv-out, compressed-out, mem2mem.

But a 'tv' node might also be able to handle compressed video (depending on how the hardware is organized), so how do you handle that?

Well, What you've called as "compressed" is, in IMO, "encoder". It probably makes sense to have, also "decoder".

I couldn't remember the name :-)

Quoted 3 lines

I'm in doubt about "webcam", as there are some
grabber devices with analog camera inputs for video surveillance. Maybe "camera" is a better name for it.

Hmm. 'webcam' or 'camera' implies settings like exposure, etc. Many video surveillance devices are just frame grabbers to which you can attach a camera, but you can just as easily attach any composite/S-video input.

Quoted 6 lines

Quoted 3 lines

It can all
be solved, I'm sure, but I'm not sure if such userspace mappings will scale that well with the increasing hardware complexity.

Not all video nodes would need an alias. Just the ones where it makes sense for an application to open it.

I'm not certain you will solve that much with this. Most people (i.e. the average linux users) have only one or two video devices, most likely a webcam and perhaps some DVB/V4L USB stick. Generic apps that needs to enumerate all devices will still need to use sysfs or go through all video nodes.

It's why I like the MC: just one device node per hardware unit. Easy to enumerate, easy to present to the user.

I'm tempted to see if I can make a proof-of-concept... Time is a problem for me, though.

Regards,

Hans

Em 26-08-2011 12:29, Hans Verkuil escreveu:

Quoted 67 lines · On Friday, August 26, 2011 17:09:02 Mauro Carvalho Chehab wrote:

Quoted 65 lines

Em 26-08-2011 11:16, Hans Verkuil escreveu:

Quoted 41 lines · On Friday, August 26, 2011 15:45:30 Laurent Pinchart wrote:

Quoted 34 lines · On Thursday 25 August 2011 14:43:56 Mauro Carvalho Chehab wrote:

Quoted 1 line

Em 24-08-2011 19:29, Sakari Ailus escreveu:

[snip]

Quoted 22 lines

Quoted 8 lines

The question I still have on this is that how should the user know which video node to access on an embedded system with a camera: the OMAP 3 ISP, for example, contains some eight video nodes which have different ISP blocks connected to them. Likely two of these nodes are useful for a general purpose application based on which image format it requests. It would make sense to provide generic applications information only on those devices they may meaningfully use.

IMO, we should create a namespace device mapping for video devices. What I

mean is that we should keep the "raw" V4L2 devices as:

	/dev/video??

But also recommend the creation of a new userspace map, like:

	/dev/webcam??
	/dev/tv??
	...

with is an alias for the actual device.

Quoted 7 lines

Something similar to dvd/cdrom aliases that already happen on most distros:

lrwxrwxrwx   1 root root           3 Ago 24 12:14 cdrom -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 cdrw -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 dvd -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 dvdrw -> sr0

Quoted 5 lines

I've been toying with a similar idea. libv4l currently wraps /dev/video* device nodes and assumes a 1:1 relationship between a video device node and a video device. Should this assumption be somehow removed, replaced by a video device concept that wouldn't be tied to a single video device node ?

Quoted 4 lines

Just as background information: the original idea was always that all v4l drivers would have a MC and that libv4l would use the information contained there as a helper (such as deciding which nodes would be the 'default' nodes for generic applications).

This is something that libv4l won't do: it is up to the userspace application to choose the device node to open. Ok, libv4l can have helper APIs for that, like the one I wrote, but even adding MC support on it may not solve the issues.

Quoted 6 lines

Since there is only one MC device node for each piece of video hardware that would make it much easier to discover what hardware there is and what video nodes to use.

I always liked that idea, although I know Mauro is opposed to having a MC for all v4l drivers.

It doesn't make sense to add MC for all V4L drivers. Not all devices are like ivtv with lots of device drivers. In a matter of fact, most supported devices create just one video node. Adding MC support for those devices will just increase the drivers complexity without _any_ reason, as those devices are fully configurable using the existing ioctl's.

Quoted 1 line

It's for consistency so applications know what to expect.

I've already said it before: We won't be implementing an API for a device just for "consistency" without any real reason.

Quoted 4 lines

For all the simple drivers you'd just need some simple core support to add a MC. What I always thought would be handy is for applications to just iterate over all MCs and show which video/dvb/audio hardware the user has in its system.

MC doesn't work for audio yet,

as snd-usb-audio doesn't use it. So, it will fail for a large amount of devices whose audio is implemented using standard USB Audio Class. Adding MC support for it doesn't sound trivial, and won't offer any gain over the sysfs equivalent.

Quoted 12 lines

Quoted 11 lines

Also, as I said before, and implemented at xawtv and at a v4l-utils library, the code may use sysfs for simpler devices. It shouldn't be hard to implement a mc aware code there, although I don't think that MC API is useful to discover what nodes are meant to be used for TV, encoder, decoder, webcams, etc. The only type information it currently provides is:

#define MEDIA_ENT_T_DEVNODE_V4L (MEDIA_ENT_T_DEVNODE + 1)
#define MEDIA_ENT_T_DEVNODE_FB (MEDIA_ENT_T_DEVNODE + 2)
#define MEDIA_ENT_T_DEVNODE_ALSA (MEDIA_ENT_T_DEVNODE + 3)
#define MEDIA_ENT_T_DEVNODE_DVB (MEDIA_ENT_T_DEVNODE + 4)

Quoted 3 lines

That's because we never added meta information like that. As long as the MC is only used for SoC/complex drivers there is no point in adding such info.

Even For SoC, such info would probably be useful.

As I said before, with the current way, I can't see how a generic MC aware application would do the right thing for the devices that currently implement the MC api, simply because there's no way for an application to know what pipelines need to be configured, as no entity at the MC (or elsewhere on some userspace library) describes what pipelines are meant to be used by the common usecase.

Quoted 1 line

It would be trivial to add precisely this type of information, though.

Yes, adding a new field to indicate what type of V4L devnode is there won't be hard, but it would be better to replace the "MEDIA_ENT_T_DEVNODE_V4L" by something that actually describes the device type.

The same kind of logic might also applies also to other device types. For example, ALSA means a playback, a capture or a mixer device? Or does it mean the complete audio hardware? (probably not, as it would hide alsa internal pipelines).

Quoted 31 lines

Quoted 5 lines

So, a MC aware application also needs to be a hardware-dependent application, as it will need to use something else, like the media entity name, to discover for what purpose a media node is meant to be used.

Quoted 7 lines

Quoted 2 lines

While I am not opposed to creating such userspace maps I also think it is a bit of a poor-man's solution.

The creation of per-type devices is part of the current API: radio and vbi nodes are examples of that (except that they aren't aliases, but real devices, but the idea is the same: different names for different types of usage).

That's why I'm not opposed to it. I'm just not sure how detailed/extensive that mapping should be.

Quoted 6 lines

Quoted 6 lines

In particular I am worried that we get a
lot of those mappings (just think of ivtv with its 8 or 9 devices).

I can think of: webcam, tv, compressed (mpeg), tv-out, compressed-out, mem2mem.

Quoted 6 lines

Quoted 3 lines

But a 'tv' node might also be able to handle compressed video (depending on how the hardware is organized), so how do you handle that?

Well, What you've called as "compressed" is, in IMO, "encoder". It probably makes sense to have, also "decoder".

I couldn't remember the name :-)

Quoted 7 lines

Quoted 3 lines

I'm in doubt about "webcam", as there are some
grabber devices with analog camera inputs for video surveillance. Maybe "camera" is a better name for it.

Hmm. 'webcam' or 'camera' implies settings like exposure, etc. Many video surveillance devices are just frame grabbers to which you can attach a camera, but you can just as easily attach any composite/S-video input.

"grabber" is for sure another type.

The thing with the device type at the name is that S_INPUT may actually change the type of device.

Btw, a proper implementation for the MC API for a device that has audio/video muxes internally would require to map the internal pipelines via MC. So, it is something that just adding a V4L core "glue" won't work. So, even a "simple" device like em28xx or bttv would require lots of changes internally, as those things are currently not implemented fully as sub-devices. It will also create a conflict at userspace, as the same pipeline could be changed via two different API's. As I said before, too much changes for no benefit.

Quoted 12 lines

Quoted 5 lines

Quoted 2 lines

It can all be solved, I'm sure, but I'm not sure if such userspace mappings will scale that well with the increasing hardware complexity.

Not all video nodes would need an alias. Just the ones where it makes sense for an application to open it.

I'm not certain you will solve that much with this. Most people (i.e. the average linux users) have only one or two video devices, most likely a webcam and perhaps some DVB/V4L USB stick. Generic apps that needs to enumerate all devices will still need to use sysfs or go through all video nodes.

Quoted 2 lines

It's why I like the MC: just one device node per hardware unit. Easy to enumerate, easy to present to the user.

It is not one device node per hardware unit. Sysfs is needed to match snd-usb-audio with the corresponding video device.

Implementing MC to cover the non-SoC cases may work after lots of efforts. It is like implementing a nuclear bomb to kill a bug. It will work, but a bug spray insecticide will produce the same effect with a very small cost to implement.

Quoted 2 lines

I'm tempted to see if I can make a proof-of-concept... Time is a problem for me, though.

Hi Mauro,

Quoted 68 lines · On Friday 26 August 2011 19:32:30 Mauro Carvalho Chehab wrote:

Em 26-08-2011 12:29, Hans Verkuil escreveu:

Quoted 67 lines · On Friday, August 26, 2011 17:09:02 Mauro Carvalho Chehab wrote:

Quoted 65 lines

Em 26-08-2011 11:16, Hans Verkuil escreveu:

Quoted 41 lines · On Friday, August 26, 2011 15:45:30 Laurent Pinchart wrote:

Quoted 34 lines · On Thursday 25 August 2011 14:43:56 Mauro Carvalho Chehab wrote:

Quoted 1 line

Em 24-08-2011 19:29, Sakari Ailus escreveu:

[snip]

Quoted 22 lines

Quoted 8 lines

The question I still have on this is that how should the user know which video node to access on an embedded system with a camera: the OMAP 3 ISP, for example, contains some eight video nodes which have different ISP blocks connected to them. Likely two of these nodes are useful for a general purpose application based on which image format it requests. It would make sense to provide generic applications information only on those devices they may meaningfully use.

IMO, we should create a namespace device mapping for video devices. What I

mean is that we should keep the "raw" V4L2 devices as:

	/dev/video??

But also recommend the creation of a new userspace map, like:

	/dev/webcam??
	/dev/tv??
	...

with is an alias for the actual device.

Quoted 7 lines

Something similar to dvd/cdrom aliases that already happen on most distros:

lrwxrwxrwx   1 root root           3 Ago 24 12:14 cdrom -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 cdrw -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 dvd -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 dvdrw -> sr0

Quoted 5 lines

I've been toying with a similar idea. libv4l currently wraps /dev/video* device nodes and assumes a 1:1 relationship between a video device node and a video device. Should this assumption be somehow removed, replaced by a video device concept that wouldn't be tied to a single video device node ?

Quoted 4 lines

Just as background information: the original idea was always that all v4l drivers would have a MC and that libv4l would use the information contained there as a helper (such as deciding which nodes would be the 'default' nodes for generic applications).

This is something that libv4l won't do: it is up to the userspace application to choose the device node to open. Ok, libv4l can have helper APIs for that, like the one I wrote, but even adding MC support on it may not solve the issues.

Quoted 6 lines

Since there is only one MC device node for each piece of video hardware that would make it much easier to discover what hardware there is and what video nodes to use.

I always liked that idea, although I know Mauro is opposed to having a MC for all v4l drivers.

It doesn't make sense to add MC for all V4L drivers. Not all devices are like ivtv with lots of device drivers. In a matter of fact, most supported devices create just one video node. Adding MC support for those devices will just increase the drivers complexity without _any_ reason, as those devices are fully configurable using the existing ioctl's.

Quoted 4 lines

Quoted 1 line

It's for consistency so applications know what to expect.

I've already said it before: We won't be implementing an API for a device just for "consistency" without any real reason.

We have a real reason: providing a single API through which applications can enumerate devices and retrieve information such as device capabilities and relationships between devices.

Quoted 7 lines

Quoted 5 lines

For all the simple drivers you'd just need some simple core support to add a MC. What I always thought would be handy is for applications to just iterate over all MCs and show which video/dvb/audio hardware the user has in its system.

MC doesn't work for audio yet,

*yet*

Quoted 21 lines

as snd-usb-audio doesn't use it. So, it will fail for a large amount of devices whose audio is implemented using standard USB Audio Class. Adding MC support for it doesn't sound trivial, and won't offer any gain over the sysfs equivalent.

Quoted 15 lines

Quoted 11 lines

Also, as I said before, and implemented at xawtv and at a v4l-utils library, the code may use sysfs for simpler devices. It shouldn't be hard to implement a mc aware code there, although I don't think that MC API is useful to discover what nodes are meant to be used for TV, encoder, decoder, webcams, etc. The only type information it currently provides is:

#define MEDIA_ENT_T_DEVNODE_V4L (MEDIA_ENT_T_DEVNODE + 1)
#define MEDIA_ENT_T_DEVNODE_FB (MEDIA_ENT_T_DEVNODE + 2)
#define MEDIA_ENT_T_DEVNODE_ALSA (MEDIA_ENT_T_DEVNODE + 3)
#define MEDIA_ENT_T_DEVNODE_DVB (MEDIA_ENT_T_DEVNODE + 4)

That's because we never added meta information like that. As long as the MC is only used for SoC/complex drivers there is no point in adding such info.

Quoted 1 line

Even For SoC, such info would probably be useful.

I agree, that's something we need. Most MC-related develop have been driven by products so far, this is just something we haven't implemented yet because there was no real need. It has always been planned though.

Quoted 7 lines

As I said before, with the current way, I can't see how a generic MC aware application would do the right thing for the devices that currently implement the MC api, simply because there's no way for an application to know what pipelines need to be configured, as no entity at the MC (or elsewhere on some userspace library) describes what pipelines are meant to be used by the common usecase.

Quoted 5 lines

Quoted 1 line

It would be trivial to add precisely this type of information, though.

Yes, adding a new field to indicate what type of V4L devnode is there won't be hard, but it would be better to replace the "MEDIA_ENT_T_DEVNODE_V4L" by something that actually describes the device type.

I don't agree with that. For complex devices a video node is just a DMA engine, it's not tied to a device description as it can be connected to different pipelines with different capabilities. What we need is a way to tag an entity with several type information. For a device with a single video node and no other entity, the video node entity will be tagged with MEDIA_ENT_T_DEVNODE_V4L as well as "camera" or "webcam" for instance (this is just an example, we need to think about the names). For a device such as the OMAP3 ISP, video nodes will be tagged with MEDIA_ENT_T_DEVNODE_V4L only and other entities will be tagged with "camera sensor", "resizer", ...

Quoted 49 lines

The same kind of logic might also applies also to other device types. For example, ALSA means a playback, a capture or a mixer device? Or does it mean the complete audio hardware? (probably not, as it would hide alsa internal pipelines).

Quoted 31 lines

Quoted 5 lines

So, a MC aware application also needs to be a hardware-dependent application, as it will need to use something else, like the media entity name, to discover for what purpose a media node is meant to be used.

Quoted 7 lines

Quoted 2 lines

While I am not opposed to creating such userspace maps I also think it is a bit of a poor-man's solution.

The creation of per-type devices is part of the current API: radio and vbi nodes are examples of that (except that they aren't aliases, but real devices, but the idea is the same: different names for different types of usage).

That's why I'm not opposed to it. I'm just not sure how detailed/extensive that mapping should be.

Quoted 6 lines

Quoted 6 lines

In particular I am worried that we get a
lot of those mappings (just think of ivtv with its 8 or 9 devices).

I can think of: webcam, tv, compressed (mpeg), tv-out, compressed-out, mem2mem.

Quoted 6 lines

Quoted 3 lines

But a 'tv' node might also be able to handle compressed video (depending on how the hardware is organized), so how do you handle that?

Well, What you've called as "compressed" is, in IMO, "encoder". It probably makes sense to have, also "decoder".

I couldn't remember the name :-)

Quoted 7 lines

Quoted 3 lines

I'm in doubt about "webcam", as there are some
grabber devices with analog camera inputs for video surveillance. Maybe "camera" is a better name for it.

Hmm. 'webcam' or 'camera' implies settings like exposure, etc. Many video surveillance devices are just frame grabbers to which you can attach a camera, but you can just as easily attach any composite/S-video input.

"grabber" is for sure another type.

The thing with the device type at the name is that S_INPUT may actually change the type of device.

Quoted 8 lines

Btw, a proper implementation for the MC API for a device that has audio/video muxes internally would require to map the internal pipelines via MC. So, it is something that just adding a V4L core "glue" won't work. So, even a "simple" device like em28xx or bttv would require lots of changes internally, as those things are currently not implemented fully as sub-devices. It will also create a conflict at userspace, as the same pipeline could be changed via two different API's. As I said before, too much changes for no benefit.

You don't have to do that. You can keep the current API unchanged, and add MC support in V4L2 core to only enumerate devices and query their capabilities.

Quoted 12 lines

Quoted 12 lines

Quoted 5 lines

Quoted 2 lines

It can all be solved, I'm sure, but I'm not sure if such userspace mappings will scale that well with the increasing hardware complexity.

Not all video nodes would need an alias. Just the ones where it makes sense for an application to open it.

I'm not certain you will solve that much with this. Most people (i.e. the average linux users) have only one or two video devices, most likely a webcam and perhaps some DVB/V4L USB stick. Generic apps that needs to enumerate all devices will still need to use sysfs or go through all video nodes.

Quoted 5 lines

Quoted 2 lines

It's why I like the MC: just one device node per hardware unit. Easy to enumerate, easy to present to the user.

It is not one device node per hardware unit. Sysfs is needed to match snd-usb-audio with the corresponding video device.

No, the goal is to have a single /dev/media* node for both the audio and video devices in the webcam.

Quoted 7 lines

Implementing MC to cover the non-SoC cases may work after lots of efforts. It is like implementing a nuclear bomb to kill a bug. It will work, but a bug spray insecticide will produce the same effect with a very small cost to implement.

Quoted 2 lines

I'm tempted to see if I can make a proof-of-concept... Time is a problem for me, though.

Em 26-08-2011 11:16, Hans Verkuil escreveu:

Quoted 40 lines · On Friday, August 26, 2011 15:45:30 Laurent Pinchart wrote:

Quoted 38 lines · On Thursday 25 August 2011 14:43:56 Mauro Carvalho Chehab wrote:

Quoted 1 line

Em 24-08-2011 19:29, Sakari Ailus escreveu:

[snip]

Quoted 21 lines

Quoted 7 lines

The question I still have on this is that how should the user know which video node to access on an embedded system with a camera: the OMAP 3 ISP, for example, contains some eight video nodes which have different ISP blocks connected to them. Likely two of these nodes are useful for a general purpose application based on which image format it requests. It would make sense to provide generic applications information only on those devices they may meaningfully use.

IMO, we should create a namespace device mapping for video devices. What I

mean is that we should keep the "raw" V4L2 devices as:

	/dev/video??

But also recommend the creation of a new userspace map, like:

	/dev/webcam??
	/dev/tv??
	...

with is an alias for the actual device.

Quoted 7 lines

Something similar to dvd/cdrom aliases that already happen on most distros:

lrwxrwxrwx   1 root root           3 Ago 24 12:14 cdrom -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 cdrw -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 dvd -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 dvdrw -> sr0

I've been toying with a similar idea. libv4l currently wraps /dev/video* device nodes and assumes a 1:1 relationship between a video device node and a video device. Should this assumption be somehow removed, replaced by a video device concept that wouldn't be tied to a single video device node ?

Quoted 4 lines

Just as background information: the original idea was always that all v4l drivers would have a MC and that libv4l would use the information contained there as a helper (such as deciding which nodes would be the 'default' nodes for generic applications).

This is something that libv4l won't do: it is up to the userspace application to choose the device node to open.

Ok, libv4l can have helper APIs for that, like the one I wrote, but even adding MC support on it may not solve the issues.

Quoted 4 lines

Since there is only one MC device node for each piece of video hardware that would make it much easier to discover what hardware there is and what video nodes to use.

Quoted 2 lines

I always liked that idea, although I know Mauro is opposed to having a MC for all v4l drivers.

It doesn't make sense to add MC for all V4L drivers. Not all devices are like ivtv with lots of device drivers. In a matter of fact, most supported devices create just one video node. Adding MC support for those devices will just increase the drivers complexity without _any_ reason, as those devices are fully configurable using the existing ioctl's.

Also, as I said before, and implemented at xawtv and at a v4l-utils library, the code may use sysfs for simpler devices. It shouldn't be hard to implement a mc aware code there, although I don't think that MC API is useful to discover what nodes are meant to be used for TV, encoder, decoder, webcams, etc. The only type information it currently provides is:

#define MEDIA_ENT_T_DEVNODE_V4L (MEDIA_ENT_T_DEVNODE + 1)
#define MEDIA_ENT_T_DEVNODE_FB (MEDIA_ENT_T_DEVNODE + 2)
#define MEDIA_ENT_T_DEVNODE_ALSA (MEDIA_ENT_T_DEVNODE + 3)
#define MEDIA_ENT_T_DEVNODE_DVB (MEDIA_ENT_T_DEVNODE + 4)

So, a MC aware application also needs to be a hardware-dependent application, as it will need to use something else, like the media entity name, to discover for what purpose a media node is meant to be used.

Quoted 2 lines

While I am not opposed to creating such userspace maps I also think it is a bit of a poor-man's solution.

The creation of per-type devices is part of the current API: radio and vbi nodes are examples of that (except that they aren't aliases, but real devices, but the idea is the same: different names for different types of usage).

Quoted 8 lines

In particular I am worried that we get a lot of those mappings (just think of ivtv with its 8 or 9 devices).

I can think of: webcam, tv, compressed (mpeg), tv-out, compressed-out, mem2mem.

But a 'tv' node might also be able to handle compressed video (depending on how the hardware is organized), so how do you handle that?

Well, What you've called as "compressed" is, in IMO, "encoder". It probably makes sense to have, also "decoder". I'm in doubt about "webcam", as there are some grabber devices with analog camera inputs for video surveillance. Maybe "camera" is a better name for it.

Quoted 2 lines

It can all be solved, I'm sure, but I'm not sure if such userspace mappings will scale that well with the increasing hardware complexity.

Not all video nodes would need an alias. Just the ones where it makes sense for an application to open it.

Em 24-08-2011 19:29, Sakari Ailus escreveu:

Quoted 21 lines

Hi Mauro,

Quoted 18 lines · Mauro Carvalho Chehab

Em 20-08-2011 04:27, Sylwester Nawrocki escreveu:

Quoted 17 lines

Hi Mauro,

Quoted 13 lines · Mauro Carvalho Chehab

It seems that there are too many miss understandings or maybe we're just talking the same thing on different ways.

So, instead of answering again, let's re-start this discussion on a different way.

One of the requirements that it was discussed a lot on both mailing lists and on the Media Controllers meetings that we had (or, at least in the ones where I've participated) is that:

"A pure V4L2 userspace application, knowing about video device
nodes only, can still use the driver. Not all advanced features will be available."

Quoted 8 lines

Quoted 4 lines

Quoted 1 line

What does a term "a pure V4L2 userspace application" mean here ?

The above quotation are exactly the Laurent's words that I took from one of his replies.

I would define this as an application which uses V4L2 but does not use Media controller or the V4L2 subdev interfaces nor is aware of any particular hardware device.

As a general rule, applications should not be aware of any particular hardware. That's why we provide standard ways of doing things. Experience shows that hardware aware applications become obsolete very fast. If you seek at the net, you'll see application-aware tools for bttv, zoran, etc. I doubt that they would still work today, as they were dependent on some particular special case driver behavior that has changed among the time. Also, if you look of the updates timeline for those, you'll see that they were kept maintained for a short period of time.

So, I think that all hardware aware dependency should be at the driver and at the libv4l only. As the proper usage of the MC API requires a hardware aware knowledge, I don't think that a generic application should bother to implement the MC API at all.

It should be noticed, however, that having a hardware/driver aware libv4l also implies that libv4l should be dependent on an specific kernel version, at the distributions.

This already happens somehow there, but, currently, a new version of libv4l can work with an older kernel (as all we currently have there is support for new FOURCC formats and quirk lists of sensors mounted upside down).

So, a version made to work with kernel 3.0 will for sure support all webcams found at kernel 2.39.

Quoted 15 lines

Quoted 6 lines

Quoted 2 lines

Does it also account an application which is linked to libv4l2 and uses calls specific to a particular hardware which are included there?

That's a good question. We need to properly define what it means, to avoid having libv4l abuses.

Quoted 8 lines

In other words, it seems ok to use libv4l to set pipelines via the MC API at open(), but it isn't ok to have an open() binary only libv4l plugin that will hook open and do the complete device initialization on userspace (I remember that one vendor once proposed a driver like that).

Also, from my side, I'd like to see both libv4l and kernel drivers being submitted together, if the new driver depends on a special libv4l support for it to work.

Quoted 8 lines

I agree with the above.

I do favour using libv4l to do the pipeline setup using MC and V4L2 subdev interfaces. This has the benefit that the driver provides just one interface to access different aspects of it, be it pipeline setup (Media controller), a control to change exposure time (V4L2 subdev) or queueing video buffer (V4L2). This means more simple and more maintainable drivers and less bugs in general.

I agree.

Quoted 3 lines

Apart from what the drivers already provide on video nodea, to support a general purpose V4L2 application, libv4l plugin can do is (not exhaustive list):

IMO, we need to write a full list of what should be done at libv4l, as it seems that there are different opinions about what should be at the driver, and what should be outside it.

Quoted 8 lines

- Perform pipeline setup using MC interface, possibly based on input selected using S_INPUT so that e.g. multiple sensors can be supported and
- implement {S,G,TRY}_EXT_CTRLS and QUERYCTRL using V4L2 subdev nodes as backend.

As the Media controller and V4L2 interfaces are standardised, I see no reason why this plugin could not be fully generic: only the configuration is device specific.

I don't think you can do everything that is needed on a fully generic plugin. 3A algorithm implementations, for example, seems to be device specific.

Of course, the maximum we can do to have a generic implementation that fits on most cases, the better.

Quoted 16 lines

This configuration could be stored into a configuration file which is selected based on the system type. On embedded systems (ARMs at least, but anyway the vast majority is based on ARM) the board type is easily available for the user space applications in /proc/cpuinfo --- this example is from the Nokia N900:

---
Processor : ARMv7 Processor rev 3 (v7l)
BogoMIPS : 249.96
Features : swp half fastmult vfp edsp neon vfpv3
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x1
CPU part : 0xc08
CPU revision : 3

Quoted 7 lines

Hardware : Nokia RX-51 board
Revision : 2101
Serial : 0000000000000000
---

I think this would be a first step to support general purpose application on embedded systems.

Agreed.

Quoted 7 lines

The question I still have on this is that how should the user know which video node to access on an embedded system with a camera: the OMAP 3 ISP, for example, contains some eight video nodes which have different ISP blocks connected to them. Likely two of these nodes are useful for a general purpose application based on which image format it requests. It would make sense to provide generic applications information only on those devices they may meaningfully use.

IMO, we should create a namespace device mapping for video devices. What I mean is that we should keep the "raw" V4L2 devices as:

	/dev/video??
But also recommend the creation of a new userspace map, like:
	/dev/webcam??
	/dev/tv??
	...

with is an alias for the actual device.

Something similar to dvd/cdrom aliases that already happen on most distros:

lrwxrwxrwx   1 root root           3 Ago 24 12:14 cdrom -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 cdrw -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 dvd -> sr0
lrwxrwxrwx   1 root root           3 Ago 24 12:14 dvdrw -> sr0

Quoted 10 lines

Later on, more functionality could be added to better support hardware which supports e.g. different pixel formats, image sizes, scaling and crop. I'm not entirely certain if all of this is fully generic --- but I think the vast majority of it is --- at least converting from v4l2_mbus_framefmt pixelcode to v4l2_format.fmt is often quite hardware specific which must be taken into account by the generic plugin.

At that point many policy decisions must be made on how to use the hardware the best way, and that must be also present in the configuration file.

Quoted 4 lines

But perhaps I'm going too far with this now; we don't yet have a generic plugin providing basic functionality. We have the OMAP 3 ISP libv4l plugin which might be a good staring point for this work.

We can start with that plugin making it more generic or forking it into two
plugins: a generic one, and an OMAP3 specific implementation for the things that are hardware-specific.

Em 28-07-2011 19:57, Sylwester Nawrocki escreveu:

Quoted 9 lines · Mauro Carvalho Chehab

Quoted 7 lines

Accumulating sub-dev controls at the video node is the right thing to do.

An MC-aware application will need to handle with that, but that doesn't sound to be hard. All such application would need to do is to first probe the subdev controls, and, when parsing the videodev controls, not register controls with duplicated ID's, or to mark them with some special attribute.

Quoted 5 lines

IMHO it's not a big issue in general. Still, both subdev and the host device may support same control id. And then even though the control ids are same on the subdev and the host they could mean physically different controls (since when registering a subdev at the host driver the host's controls take precedence and doubling subdev controls are skipped).

True, but, except for specific usecases, the host control is enough.

Quoted 4 lines

Also there might be some preference at user space, at which stage of the pipeline to apply some controls. This is where the subdev API helps, and plain video node API does not.

Again, this is for specific usecases. On such cases, what is expected is that the more generic control will be exported via V4L2 API.

Quoted 12 lines

Quoted 11 lines

Quoted 4 lines

Thus it's a bit hard to imagine that we could do something like "optionally not to inherit controls" as the subdev/MC API is optional. :)

This was actually implemented. There are some cases at ivtv/cx18 driver where both the bridge and a subdev provides the same control (audio volume, for example). The idea is to allow the bridge driver to touch at the subdev control without exposing it to userspace, since the desire was that the bridge driver itself would expose such control, using a logic that combines changing the subdev and the bridge registers for volume.

Quoted 3 lines

This seem like hard coding a policy in the driver;) Then there is no way (it might not be worth the effort though) to play with volume level at both devices, e.g. to obtain optimal S/N ratio.

In general, playing with just one control is enough. Andy had a different opinion when this issue were discussed, and he thinks that playing with both is better. At the end, this is a developers decision, depending on how much information (and bug reports) he had.

Quoted 4 lines

This is a hack...sorry, just joking ;-) Seriously, I think the situation with the userspace subdevs is a bit different. Because with one API we directly expose some functionality for applications, with other we code it in the kernel, to make the devices appear uniform at user space.

Not sure if I understood you. V4L2 export drivers functionality to userspace in an uniform way. MC api is for special applications that might need to access some internal functions on embedded devices.

Of course, there are some cases where it doesn't make sense to export a subdev control via V4L2 API.

Quoted 10 lines

Quoted 9 lines

Quoted 4 lines

Also, the sensor subdev can be configured in the video node driver as well as through the subdev device node. Both APIs can do the same thing but in order to let the subdev API work as expected the video node driver must be forbidden to configure the subdev.

Why? For the sensor, a V4L2 API call will look just like a bridge driver call. The subdev will need a mutex anyway, as two MC applications may be opening it simultaneously. I can't see why it should forbid changing the control from the bridge driver call.

Quoted 9 lines

Please do not forget there might be more than one subdev to configure and that the bridge itself is also a subdev (which exposes a scaler interface, for instance). A situation pretty much like in Figure 4.4 [1] (after the scaler there is also a video node to configure, but we may assume that pixel resolution at the scaler pad 1 is same as at the video node). Assuming the format and crop configuration flow is from sensor to host scaler direction, if we have tried to configure _all_ subdevs when the last stage of the pipeline is configured (i.e. video node) the whole scaler and crop/composition configuration we have been destroyed at that time. And there is more to configure than VIDIOC_S_FMT can do.

Think from users perspective: all user wants is to see a video of a given resolution. S_FMT (and a few other VIDIOC_* calls) have everything that the user wants: the desired resolution, framerate and format.

Specialized applications indeed need more, in order to get the best images for certain types of usages. So, MC is there.

Such applications will probably need to know exactly what's the sensor, what are their bugs, how it is connected, what are the DSP blocks in the patch, how the DSP algorithms are implemented, etc, in order to obtain the the perfect image.

Even on embedded devices like smartphones and tablets, I predict that both types of applications will be developed and used: people may use a generic application like flash player, and an specialized application provided by the manufacturer. Users can even develop their own applications generic apps using V4L2 directly, at the devices that allow that.

As I said before: both application types are welcome. We just need to warrant that a pure V4L application will work reasonably well.

Quoted 3 lines

Allowing the bridge driver to configure subdevs at all times would prevent the subdev/MC API to work.

Well, then we need to think on an alternative for that. It seems an interesting theme for the media workshop at the Kernel Summit/2011.

Quoted 15 lines

Quoted 14 lines

Quoted 3 lines

There is a conflict there that in order to use 'optional' API the 'main' API behaviour must be affected....

It is optional from userspace perspective. A V4L2-only application should be able to work with all drivers. However, a MC-aware application will likely be specific for some hardware, as it will need to know some device-specific stuff.

Both kinds of applications are welcome, but dropping support for V4L2-only applications is the wrong thing to do.

Quoted 1 line

And I really cant use V4L2 API only as is because it's too limited.

Why?

Quoted 4 lines

For instance there is really yet no support for scaler and composition onto a target buffer in the Video Capture Interface (we also use sensors with built in scalers). It's difficult to efficiently manage capture/preview pipelines. It is impossible to discover the system topology.

Scaler were always supported by V4L2: if the resolution specified by S_FMT is not what the sensor provides, then scale. All non-embedded drivers with sensor or bridge supports scale does that.

Composition is not properly supported yet. It could make sense to add it to V4L. How do you think MC API would help with composite?

Managing capture/preview pipelines will require some support at V4L2 level. This is a problem that needs to be addressed there anyway, as buffers for preview/capture need to be allocated. There's an RFC about that, but I don't think it covers the pipelines for it.

Discovering the system topology indeed is not part of V4L2 API and will never be. This is MC API business. There's no overlap with V4L2.

Quoted 9 lines

Quoted 5 lines

Quoted 2 lines

Might be that's why I see more and more often migration to OpenMAX recently.

I don't think so. People may be adopting OpenMAX just because of some marketing strategy from the OpenMAX forum. We don't spend money to announce V4L2 ;)

:)

Quoted 1 line

Quoted 7 lines

Quoted 3 lines

I think that writing a pure OpenMAX driver is the wrong thing to do, as, at the long term, it will cost _a_lot_ for the vendors to maintain something that will never be merged upstream.

In general it depends on priorities. For some chip vendors it might be more important to provide a solution in short time frame. Getting something in mainline isn't effortless and spending lot's of time on this for some parties is unacceptable.

Adding something at a forum like OpenMAX is probably not an easy task ;) It generally takes a long time to change something on open forum specifications. Also, participating on those forums require lots of money, with membership and travel expenses.

Adding a new driver that follows the API's is not a long-term effort. The delay is basically one kernel cycle, e. g. about 3-4 months.

Most of the delays on merging drivers for embedded systems, however, take a way longer than that, unfortunately. From what I see, what is delaying driver submission is due to:
- the lack of how to work with the Linux community. Some developers take a
long time to get the 'modus operandi';
- the need of API changes. It is still generally faster to add a new API
addition at the Kernel than on an open forum;
- the discussions inside the various teams (generally from the same company,
or the company and their customers) about the better way to implement some feature.

All the above also happens when developing an OpenMAX driver: companies need to learn how to work with the OpenMax forums, API changes may be required, so they need to participate at the forums, the several internal teams and customers will be discussing the requirements.

I bet that there's also one additional step: to submit the driver to some company that will check the driver compliance with the official API. Such certification process is generally expensive and takes a long time.

At the end of the day, they'll also spend a lot of time to have the driver done, or they'll end by releasing a "not-quite-openmax" driver, and then needing to rework on that, due to customers complains, or to certification-compliance, loosing time and money.

From what I can see many of the big players in the graphics devices world are already

Hi Sylwester,

Em 27-07-2011 13:35, Sylwester Nawrocki escreveu:

Quoted 31 lines

Hi Mauro,

The following changes since commit f0a21151140da01c71de636f482f2eddec2840cc:

Merge tag 'v3.0' into staging/for_v3.1 (2011-07-22 13:33:14 -0300)

are available in the git repository at:

git://git.infradead.org/users/kmpark/linux-2.6-samsung fimc-for-mauro

Sylwester Nawrocki (28):
s5p-fimc: Add support for runtime PM in the mem-to-mem driver
s5p-fimc: Add media entity initialization
s5p-fimc: Remove registration of video nodes from probe()
s5p-fimc: Remove sclk_cam clock handling
s5p-fimc: Limit number of available inputs to one
s5p-fimc: Remove sensor management code from FIMC capture driver
s5p-fimc: Remove v4l2_device from video capture and m2m driver
s5p-fimc: Add the media device driver
s5p-fimc: Conversion to use struct v4l2_fh
s5p-fimc: Conversion to the control framework
s5p-fimc: Add media operations in the capture entity driver
s5p-fimc: Add PM helper function for streaming control
s5p-fimc: Correct color format enumeration
s5p-fimc: Convert to use media pipeline operations
s5p-fimc: Add subdev for the FIMC processing block
s5p-fimc: Add support for camera capture in JPEG format
s5p-fimc: Add v4l2_device notification support for single frame capture
s5p-fimc: Use consistent names for the buffer list functions
s5p-fimc: Add runtime PM support in the camera capture driver
s5p-fimc: Correct crop offset alignment on exynos4

Quoted 8 lines

s5p-fimc: Remove single-planar capability flags
noon010pc30: Do not ignore errors in initial controls setup
noon010pc30: Convert to the pad level ops
noon010pc30: Clean up the s_power callback
noon010pc30: Remove g_chip_ident operation handler
s5p-csis: Handle all available power supplies
s5p-csis: Rework of the system suspend/resume helpers
s5p-csis: Enable v4l subdev device node

From the last time you've submitted a similar set of patches:

Quoted 9 lines

Quoted 1 line

Why? The proper way to select an input is via S_INPUT. The driver

may also

Quoted 1 line

optionally allow changing it via the media device, but it should

not be

Quoted 1 line

a mandatory requirement, as the media device API is optional.

The problem I'm trying to solve here is sharing the sensors and mipi-csi receivers between multiple FIMC H/W instances. Previously the driver supported attaching a sensor to only one selected FIMC at compile time. You could, for instance, specify all sensors as the selected FIMC's platform data and then use S_INPUT to choose between them. The sensor could not be used together with any other FIMC. But this is desired due to different capabilities of the FIMC IP instances. And now, instead of hardcoding a sensor assigment to particular video node, the sensors are bound to the media device. The media device driver takes the list of sensors and attaches them one by one to subsequent FIMC instances when it is initializing. Each sensor has a link to each FIMC but only one of them is active by default. That said an user application can use selected camera by opening corresponding video node. Which camera is at which node can be queried with G_INPUT.

I could try to implement the previous S_INPUT behaviour, but IMHO this would lead to considerable and unnecessary driver code complication due to supporting overlapping APIs

From this current pull request:

From c6fb462c38be60a45d16a29a9e56c886ee0aa08c Mon Sep 17 00:00:00 2001
From: Sylwester Nawrocki <[email protected]>
Date: Fri, 10 Jun 2011 20:36:51 +0200
Subject: s5p-fimc: Conversion to the control framework
Cc: Linux Media Mailing List <[email protected]>

Make the driver inherit sensor controls when video node only API compatibility mode is enabled. The control framework allows to properly handle sensor controls through controls inheritance when pipeline is re-configured at media device level.

...

(quoted diff) +0 -3

-       .vidioc_queryctrl               = fimc_vidioc_queryctrl,
-       .vidioc_g_ctrl                  = fimc_vidioc_g_ctrl,
-       .vidioc_s_ctrl                  = fimc_cap_s_ctrl,
...

I'll need to take some time to review this patchset. So, it will likely miss the bus for 3.1.

While the code inside this patch looked ok, your comments scared me ;)

In summary: The V4L2 API is not a legacy API that needs a "compatibility mode". Removing controls like VIDIOC_S_INPUT, VIDIOC_*CTRL, etc in favor of the media controller API is wrong. This specific patch itself seems ok, but it is easy to loose the big picture on a series of 28 patches with about 4000 lines changed.