Skip to contentSkip to conversation

Sensors and similar - subsystem interactions, divisions, bindings etc.

Ksummit-discuss Jonathan Cameron 13 participants 41 messages Jul 20, 2016 – Aug 2, 2016

lore
Share & tools
lore
#0 lore raw

Hi All,

This topic would be around the way the various subsystems interact, in the rough area of 'sensors' (I haven't yet had much of an issue with subsystem crossing with output devices but maybe that's just over the next hill!)

Scope may well be wider but includes:
* input (some of it)
* hwmon
* iio
* comedi(?)
* thermal
* power/battery
* gpio - the blur between gpios and beaglescope / PLC type I/O.
* v4l - when does a device jump from being a multi pixel thermopile to a thermal camera? Smart fingerprint scanners etc. Gesture recognition sensors (effectively 9ish pixel cameras)
* Lots of random things we haven't seen yet.

All these make use of devices that are at their hearts ADCs. Whilst it would nice if the underlying devices always fell into one clear category that is not always the case. Hardware descriptions (e.g. device tree) are an important area of contention.

The examples that follow are mostly based around IIO interactions with hwmon / input as that's my area of expertise. However, I know that similar issues occur at other boundaries.

Guenter Roeck has been working on hwmon / thermal interfacing recently. We also regularly push drivers one way or the other around that boundary.

Dmitry and I often have to decide on the IIO / input scope boundary (and we've moved it over time) + a number of cross over drivers have turned up recently.

The other subsystems I encounter have been 'quieter' in their interactions with me but many of the issues below apply to them as well.

A rough list of related topics that might be worth face to face discussion includes:

1) Are we happy with the somewhat adhoc divisions of subsystems as they stand?

I am personally happy enough with this, but it's worth checking everyone else is. This probably causes more pain for new submitters than it does for old hands. Perhaps we can come up with a checklist style doc to save people having to ask.

We have had quite a few cases where a whole driver is submitted to the 'wrong' subsystem. Not the best first interaction with the kernel world for new contributors. (This sort of area is attracts newbies as it's relatively simple and the hardware is fairly cheap and more than once their first experience with review as exactly this!)

If we aren't happy, what do we do about it?

I'm hoping we are happy or, at least, resigned to the current state of affairs, but think the question should be asked from time to time as the answer may well change + the discussion will highlight pinch points we can work on.

2) Bridging drivers - there will always be cross over cases. E.g. Generic ADCs used for battery voltage monitoring. We have a number of bridges in mainline already and others out of tree. How important are these and what features would make them more useful?

Is it ever sensible to have two drivers for the same part because the use cases are so different? (I hope there is always a better way but maybe not)

This covers both generic bridges (e.g. iio-hwmon) and device specific bridges (a good example of a touch screen driver turned up in my inbox yesterday).

3) Bindings (device tree and similar). When it is appropriate to use a device tree to describe the overlap (bridging of channels)? Sometimes there is obvious real hardware involved (a thermistor on a generic ADC input for example), but there are many grey areas.

Finally bashing out an answer to the issue device tree maintainers have with the iio-hwmon bindings would be great.
On a less biased note, that example is pretty fiddly to define but would act as a good basis to work from more generally.

We know we got it wrong (back in the dark ages), but it isn't obvious how to get it right!

IIO-input has been partly held outside mainline for years by this issue.

Also some practical design decisions to make around how to implement these mappings to work best with deferred probing etc.

4) Should we drop all the bindings for bridging between such subsystems and do it all from userspace?

I think we may end up with a hybrid of the two, but need to be able to make it work in 'standard' cases without userspace being involved. That hybrid solution may well be devicetree overlay based... Unclear so far.
+ plenty of crazy things that 'might' make sense where there isn't even the pretence of representing real hardware.

5) Complex device interaction usecases. At the moment the ones I've come across are mostly contained in IIO. The moment we start sticking in MUXes, AFEs (Analog Front Ends) and straightforward analog sensors in the mix it can get fiddly. Swapping war stories may well be worthwhile on this. This stuff also turns up in ASoC for example so probably lessons to be learned from there.

The analog devices software defined radios are another possible case study.

There of course may well be lessons to be learned from similar interactions elsewhere in the kernel.

There is a lot of history in how we ended up where we are (it all made absolute sense at the time). Sitting back and taking the time to discuss the future would be great. Whilst this might be solvable by email we've made no definitive progress for years
(and what has been made has been on a case by case basis deep in driver reviews.)

I threw comedi in the list above but, at the moment, I think the more likely direction there is a single userspace library abstracting the underlying subsystem (Analog Devices are working in that direction - perhaps Lars can offer more on that?).

GPIO is another interesting case - a lot of hardware is capable of parallel sampling, some at high speeds. It's another area that is probably too specialist for this discussion, but if people want to dive into the details it might be interesting.

I think we have only a small amount of fuzz around the v4l boundary, but wanted to leave the door open if anyone wants to discuss that one further as it's come up a few times over recent years.

The SoC world is a major case of one device, many uses. Some SoCs
are turning up with multiple ADC units, sometimes with different designs, sometimes simply so that the same hardware can be used for different things.

A few suggestions for people:

Guenter Roeck (hmwon)
Dmitry Torokhov (input)
Me (Jonathan Cameron - IIO)

Someone thermal related (not sure who would be most interested) Someone power supply related? (Guenter, any suggestions?)

Someone device tree related (Mark Rutland? Rob Herring?)

Linus Waleij (gpio)

Mark Brown - as some of concepts of IIO bridging and how far it can be taken (everything via a bridge driver) came from discussions with him on SoC ADC handling.

Lar-Peter Clausen for generally doing insane things with ADCs + having one foot in the userspace side of things.

Thanks,

Jonathan Cameron
IIO Maintainer

#1 lore raw
Quoted 5 lines · Jonathan Cameron

Hi All,

This topic would be around the way the various subsystems interact, in the rough area of 'sensors' (I haven't yet had much of an issue with subsystem crossing with output devices but maybe that's just over the next hill!)

This happens regularly with drm/kms vs v4l, especially for embedded devices where video inputs and outputs are often connected internally (i.e. paths straight from video capture to video output without needing to go to memory first).

Quoted 12 lines

Scope may well be wider but includes:
* input (some of it)
* hwmon
* iio
* comedi(?)
* thermal
* power/battery
* gpio - the blur between gpios and beaglescope / PLC type I/O.
* v4l - when does a device jump from being a multi pixel thermopile to a thermal camera? Smart fingerprint scanners etc. Gesture recognition sensors (effectively 9ish pixel cameras)

See for an example of that this patch series that adds support for v4l-touch devices:

http://www.spinics.net/lists/linux-input/msg45918.html

Quoted 45 lines

* Lots of random things we haven't seen yet.

All these make use of devices that are at their hearts ADCs. Whilst it would nice if the underlying devices always fell into one clear category that is not always the case. Hardware descriptions (e.g. device tree) are an important area of contention.

The examples that follow are mostly based around IIO interactions with hwmon / input as that's my area of expertise. However, I know that similar issues occur at other boundaries.

Guenter Roeck has been working on hwmon / thermal interfacing recently. We also regularly push drivers one way or the other around that boundary.

Dmitry and I often have to decide on the IIO / input scope boundary (and we've moved it over time) + a number of cross over drivers have turned up recently.

The other subsystems I encounter have been 'quieter' in their interactions with me but many of the issues below apply to them as well.

A rough list of related topics that might be worth face to face discussion includes:

1) Are we happy with the somewhat adhoc divisions of subsystems as they stand?

I am personally happy enough with this, but it's worth checking everyone else is. This probably causes more pain for new submitters than it does for old hands. Perhaps we can come up with a checklist style doc to save people having to ask.

We have had quite a few cases where a whole driver is submitted to the 'wrong' subsystem. Not the best first interaction with the kernel world for new contributors. (This sort of area is attracts newbies as it's relatively simple and the hardware is fairly cheap and more than once their first experience with review as exactly this!)

If we aren't happy, what do we do about it?

I'm hoping we are happy or, at least, resigned to the current state of affairs, but think the question should be asked from time to time as the answer may well change + the discussion will highlight pinch points we can work on.

Quoted 8 lines

2) Bridging drivers - there will always be cross over cases. E.g. Generic ADCs used for battery voltage monitoring. We have a number of bridges in mainline already and others out of tree. How important are these and what features would make them more useful?

Is it ever sensible to have two drivers for the same part because the use cases are so different? (I hope there is always a better way but maybe not)

One example of this is video output: both drm/kms and v4l can do video output. Video output using v4l is specifically geared towards frame-based video output whereas drm/kms is for desktop environments/GPUs. Luckily there are very few v4l video output drivers and the driver duplication occurs only for the i2c video transmitter devices. I know of two that are duplicated:
adv7511 and (I think) saa7127. The rarity is the reason nobody ever created a shim or something similar to allow for one i2c driver to serve both subsystems. Technically this would be perfectly possible, it's just work.

Quoted 31 lines

This covers both generic bridges (e.g. iio-hwmon) and device specific bridges (a good example of a touch screen driver turned up in my inbox yesterday).

3) Bindings (device tree and similar). When it is appropriate to use a device tree to describe the overlap (bridging of channels)? Sometimes there is obvious real hardware involved (a thermistor on a generic ADC input for example), but there are many grey areas.

Finally bashing out an answer to the issue device tree maintainers have with the iio-hwmon bindings would be great.
On a less biased note, that example is pretty fiddly to define but would act as a good basis to work from more generally.

We know we got it wrong (back in the dark ages), but it isn't obvious how to get it right!

IIO-input has been partly held outside mainline for years by this issue.

Also some practical design decisions to make around how to implement these mappings to work best with deferred probing etc.

4) Should we drop all the bindings for bridging between such subsystems and do it all from userspace?

I think we may end up with a hybrid of the two, but need to be able to make it work in 'standard' cases without userspace being involved. That hybrid solution may well be devicetree overlay based... Unclear so far.
+ plenty of crazy things that 'might' make sense where there isn't even the pretence of representing real hardware.

Quoted 6 lines

5) Complex device interaction usecases. At the moment the ones I've come across are mostly contained in IIO. The moment we start sticking in MUXes, AFEs (Analog Front Ends) and straightforward analog sensors in the mix it can get fiddly. Swapping war stories may well be worthwhile on this. This stuff also turns up in ASoC for example so probably lessons to be learned from there.

V4L has a long history of this: if you have multiple video scalers in your pipeline, how do you control them? E.g. there can be scalers in the video sensor and in the SoC. We came up with the media controller device to expose the topology of the hardware and to be able to control each bit directly from userspace.

While the media controller is quite powerful and can be used to expose connections across subsystems, it also requires more work from userspace, and that's an area where not enough attention has been given.

Quoted 2 lines

The analog devices software defined radios are another possible case study.

Note that the V4L2 subsystem has support for several SDR devices. Probably another area where drivers can end up in different subsystems.

Quoted 11 lines

There of course may well be lessons to be learned from similar interactions elsewhere in the kernel.

There is a lot of history in how we ended up where we are (it all made absolute sense at the time). Sitting back and taking the time to discuss the future would be great. Whilst this might be solvable by email we've made no definitive progress for years
(and what has been made has been on a case by case basis deep in driver reviews.)

Quoted 4 lines

I threw comedi in the list above but, at the moment, I think the more likely direction there is a single userspace library abstracting the underlying subsystem (Analog Devices are working in that direction - perhaps Lars can offer more on that?).

When you get complex systems you probably have to move in that direction. I'd be interesting in such work going on in other subsystems. As mentioned above, the media controller needs something like that and it would be good to learn from others.

Quoted 8 lines

GPIO is another interesting case - a lot of hardware is capable of parallel sampling, some at high speeds. It's another area that is probably too specialist for this discussion, but if people want to dive into the details it might be interesting.

I think we have only a small amount of fuzz around the v4l boundary, but wanted to leave the door open if anyone wants to discuss that one further as it's come up a few times over recent years.

There is more of that than we wish, but quite often it is just dropped in the end as being too much work to solve and not quite important enough to spend the time and money on.

Quoted 15 lines

The SoC world is a major case of one device, many uses. Some SoCs
are turning up with multiple ADC units, sometimes with different designs, sometimes simply so that the same hardware can be used for different things.

A few suggestions for people:

Guenter Roeck (hmwon)
Dmitry Torokhov (input)
Me (Jonathan Cameron - IIO)

Someone thermal related (not sure who would be most interested) Someone power supply related? (Guenter, any suggestions?)

Someone device tree related (Mark Rutland? Rob Herring?)

Quoted 8 lines

Linus Waleij (gpio)

Mark Brown - as some of concepts of IIO bridging and how far it can be taken (everything via a bridge driver) came from discussions with him on SoC ADC handling.

Lar-Peter Clausen for generally doing insane things with ADCs + having one foot in the userspace side of things.

Laurent Pinchart or myself for v4l/mc

Regards,

Hans

#2 lore raw
Quoted 27 lines · Hans Verkuil
Quoted 5 lines · Jonathan Cameron

Hi All,

This topic would be around the way the various subsystems interact, in the rough area of 'sensors' (I haven't yet had much of an issue with subsystem crossing with output devices but maybe that's just over the next hill!)

This happens regularly with drm/kms vs v4l, especially for embedded devices where video inputs and outputs are often connected internally (i.e. paths straight from video capture to video output without needing to go to memory first).

Quoted 12 lines

Scope may well be wider but includes:
* input (some of it)
* hwmon
* iio
* comedi(?)
* thermal
* power/battery
* gpio - the blur between gpios and beaglescope / PLC type I/O.
* v4l - when does a device jump from being a multi pixel thermopile to a thermal camera? Smart fingerprint scanners etc. Gesture recognition sensors (effectively 9ish pixel cameras)

Quoted 4 lines

See for an example of that this patch series that adds support for v4l-touch devices:

http://www.spinics.net/lists/linux-input/msg45918.html

Certainly interesting - never occurred to me that you could get data where that made sense from touch devices.

Quoted 55 lines
Quoted 46 lines

* Lots of random things we haven't seen yet.

All these make use of devices that are at their hearts ADCs. Whilst it would nice if the underlying devices always fell into one clear category that is not always the case. Hardware descriptions (e.g. device tree) are an important area of contention.

The examples that follow are mostly based around IIO interactions with hwmon / input as that's my area of expertise. However, I know that similar issues occur at other boundaries.

Guenter Roeck has been working on hwmon / thermal interfacing recently. We also regularly push drivers one way or the other around that boundary.

Dmitry and I often have to decide on the IIO / input scope boundary (and we've moved it over time) + a number of cross over drivers have turned up recently.

The other subsystems I encounter have been 'quieter' in their interactions with me but many of the issues below apply to them as well.

A rough list of related topics that might be worth face to face discussion includes:

1) Are we happy with the somewhat adhoc divisions of subsystems as they stand?

I am personally happy enough with this, but it's worth checking everyone else is. This probably causes more pain for new submitters than it does for old hands. Perhaps we can come up with a checklist style doc to save people having to ask.

We have had quite a few cases where a whole driver is submitted to the 'wrong' subsystem. Not the best first interaction with the kernel world for new contributors. (This sort of area is attracts newbies as it's relatively simple and the hardware is fairly cheap and more than once their first experience with review as exactly this!)

If we aren't happy, what do we do about it?

I'm hoping we are happy or, at least, resigned to the current state of affairs, but think the question should be asked from time to time as the answer may well change + the discussion will highlight pinch points we can work on.

Quoted 8 lines

2) Bridging drivers - there will always be cross over cases. E.g. Generic ADCs used for battery voltage monitoring. We have a number of bridges in mainline already and others out of tree. How important are these and what features would make them more useful?

Is it ever sensible to have two drivers for the same part because the use cases are so different? (I hope there is always a better way but maybe not)

Quoted 8 lines

One example of this is video output: both drm/kms and v4l can do video output. Video output using v4l is specifically geared towards frame-based video output whereas drm/kms is for desktop environments/GPUs. Luckily there are very few v4l video output drivers and the driver duplication occurs only for the i2c video transmitter devices. I know of two that are duplicated:
adv7511 and (I think) saa7127. The rarity is the reason nobody ever created a shim or something similar to allow for one i2c driver to serve both subsystems. Technically this would be perfectly possible, it's just work.

So the balance here (at least when we aren't doing the work) is how much maintainers push back to get this sort of work done before taking a driver. Always balances to be struck.

Quoted 45 lines
Quoted 32 lines

This covers both generic bridges (e.g. iio-hwmon) and device specific bridges (a good example of a touch screen driver turned up in my inbox yesterday).

3) Bindings (device tree and similar). When it is appropriate to use a device tree to describe the overlap (bridging of channels)? Sometimes there is obvious real hardware involved (a thermistor on a generic ADC input for example), but there are many grey areas.

Finally bashing out an answer to the issue device tree maintainers have with the iio-hwmon bindings would be great.
On a less biased note, that example is pretty fiddly to define but would act as a good basis to work from more generally.

We know we got it wrong (back in the dark ages), but it isn't obvious how to get it right!

IIO-input has been partly held outside mainline for years by this issue.

Also some practical design decisions to make around how to implement these mappings to work best with deferred probing etc.

4) Should we drop all the bindings for bridging between such subsystems and do it all from userspace?

I think we may end up with a hybrid of the two, but need to be able to make it work in 'standard' cases without userspace being involved. That hybrid solution may well be devicetree overlay based... Unclear so far.
+ plenty of crazy things that 'might' make sense where there isn't even the pretence of representing real hardware.

Quoted 6 lines

5) Complex device interaction usecases. At the moment the ones I've come across are mostly contained in IIO. The moment we start sticking in MUXes, AFEs (Analog Front Ends) and straightforward analog sensors in the mix it can get fiddly. Swapping war stories may well be worthwhile on this. This stuff also turns up in ASoC for example so probably lessons to be learned from there.

V4L has a long history of this: if you have multiple video scalers in your pipeline, how do you control them? E.g. there can be scalers in the video sensor and in the SoC. We came up with the media controller device to expose the topology of the hardware and to be able to control each bit directly from userspace.

Quoted 3 lines

While the media controller is quite powerful and can be used to expose connections across subsystems, it also requires more work from userspace, and that's an area where not enough attention has been given.

Agreed. When I've discussed this with Lars-Peter who has been looking at expanding the scope of the media control framework into IIO he has suggested that it makes a lot of sense for DSP type configurations, but perhaps not simple stuff like what is on the end of a wire...

Definitely another thing to include in any discussion though.

Quoted 33 lines
Quoted 3 lines

The analog devices software defined radios are another possible case study.

Note that the V4L2 subsystem has support for several SDR devices. Probably another area where drivers can end up in different subsystems.

Quoted 11 lines

There of course may well be lessons to be learned from similar interactions elsewhere in the kernel.

There is a lot of history in how we ended up where we are (it all made absolute sense at the time). Sitting back and taking the time to discuss the future would be great. Whilst this might be solvable by email we've made no definitive progress for years
(and what has been made has been on a case by case basis deep in driver reviews.)

Quoted 4 lines

I threw comedi in the list above but, at the moment, I think the more likely direction there is a single userspace library abstracting the underlying subsystem (Analog Devices are working in that direction - perhaps Lars can offer more on that?).

When you get complex systems you probably have to move in that direction. I'd be interesting in such work going on in other subsystems. As mentioned above, the media controller needs something like that and it would be good to learn from others.

Quoted 5 lines

GPIO is another interesting case - a lot of hardware is capable of parallel sampling, some at high speeds. It's another area that is probably too specialist for this discussion, but if people want to dive into the details it might be interesting.

Quoted 7 lines
Quoted 3 lines

I think we have only a small amount of fuzz around the v4l boundary, but wanted to leave the door open if anyone wants to discuss that one further as it's come up a few times over recent years.

There is more of that than we wish, but quite often it is just dropped in the end as being too much work to solve and not quite important enough to spend the time and money on.

Sounds like I underestimate how fiddly the boundaries between subsystems I haven't often interacted with are! Thanks for the info.

Quoted 18 lines
Quoted 18 lines

The SoC world is a major case of one device, many uses. Some SoCs
are turning up with multiple ADC units, sometimes with different designs, sometimes simply so that the same hardware can be used for different things.

A few suggestions for people:

Guenter Roeck (hmwon)
Dmitry Torokhov (input)
Me (Jonathan Cameron - IIO)

Someone thermal related (not sure who would be most interested) Someone power supply related? (Guenter, any suggestions?)

Someone device tree related (Mark Rutland? Rob Herring?)

Linus Waleij (gpio)

Quoted 8 lines
Quoted 6 lines

Mark Brown - as some of concepts of IIO bridging and how far it can be taken (everything via a bridge driver) came from discussions with him on SoC ADC handling.

Lar-Peter Clausen for generally doing insane things with ADCs + having one foot in the userspace side of things.

Laurent Pinchart or myself for v4l/mc

Cool.

Quoted 5 lines

Regards,

Hans

#3 lore raw
In reply to Hans Verkuil

Hi Hans,

Quoted 197 lines · On Thursday 21 Jul 2016 09:39:04 Hans Verkuil wrote:
Quoted 5 lines · Jonathan Cameron

Hi All,

This topic would be around the way the various subsystems interact, in the rough area of 'sensors' (I haven't yet had much of an issue with subsystem crossing with output devices but maybe that's just over the next hill!)

This happens regularly with drm/kms vs v4l, especially for embedded devices where video inputs and outputs are often connected internally (i.e. paths straight from video capture to video output without needing to go to memory first).

Quoted 11 lines

Scope may well be wider but includes:
* input (some of it)
* hwmon
* iio
* comedi(?)
* thermal
* power/battery
* gpio - the blur between gpios and beaglescope / PLC type I/O.
* v4l - when does a device jump from being a multi pixel thermopile to a thermal camera? Smart fingerprint scanners etc. Gesture recognition sensors (effectively 9ish pixel cameras)

See for an example of that this patch series that adds support for v4l-touch devices:

http://www.spinics.net/lists/linux-input/msg45918.html

Quoted 45 lines

* Lots of random things we haven't seen yet.

All these make use of devices that are at their hearts ADCs. Whilst it would nice if the underlying devices always fell into one clear category that is not always the case. Hardware descriptions (e.g. device tree) are an important area of contention.

The examples that follow are mostly based around IIO interactions with hwmon / input as that's my area of expertise. However, I know that similar issues occur at other boundaries.

Guenter Roeck has been working on hwmon / thermal interfacing recently. We also regularly push drivers one way or the other around that boundary.

Dmitry and I often have to decide on the IIO / input scope boundary (and we've moved it over time) + a number of cross over drivers have turned up recently.

The other subsystems I encounter have been 'quieter' in their interactions with me but many of the issues below apply to them as well.

A rough list of related topics that might be worth face to face discussion includes:

1) Are we happy with the somewhat adhoc divisions of subsystems as they stand?

I am personally happy enough with this, but it's worth checking everyone else is. This probably causes more pain for new submitters than it does for old hands. Perhaps we can come up with a checklist style doc to save people having to ask.

We have had quite a few cases where a whole driver is submitted to the 'wrong' subsystem. Not the best first interaction with the kernel world for new contributors. (This sort of area is attracts newbies as it's relatively simple and the hardware is fairly cheap and more than once their first experience with review as exactly this!)

If we aren't happy, what do we do about it?

I'm hoping we are happy or, at least, resigned to the current state of affairs, but think the question should be asked from time to time as the answer may well change + the discussion will highlight pinch points we can work on.

Quoted 8 lines

2) Bridging drivers - there will always be cross over cases. E.g. Generic ADCs used for battery voltage monitoring. We have a number of bridges in mainline already and others out of tree. How important are these and what features would make them more useful?

Is it ever sensible to have two drivers for the same part because the use cases are so different? (I hope there is always a better way but maybe not)

One example of this is video output: both drm/kms and v4l can do video output. Video output using v4l is specifically geared towards frame-based video output whereas drm/kms is for desktop environments/GPUs. Luckily there are very few v4l video output drivers and the driver duplication occurs only for the i2c video transmitter devices. I know of two that are
duplicated: adv7511 and (I think) saa7127. The rarity is the reason nobody ever created a shim or something similar to allow for one i2c driver to serve both subsystems. Technically this would be perfectly possible, it's just work.

Quoted 31 lines

This covers both generic bridges (e.g. iio-hwmon) and device specific bridges (a good example of a touch screen driver turned up in my inbox yesterday).

3) Bindings (device tree and similar). When it is appropriate to use a device tree to describe the overlap (bridging of channels)? Sometimes there is obvious real hardware involved (a thermistor on a generic ADC input for example), but there are many grey areas.

Finally bashing out an answer to the issue device tree maintainers have with the iio-hwmon bindings would be great.
On a less biased note, that example is pretty fiddly to define but would act as a good basis to work from more generally.

We know we got it wrong (back in the dark ages), but it isn't obvious how to get it right!

IIO-input has been partly held outside mainline for years by this issue.

Also some practical design decisions to make around how to implement these mappings to work best with deferred probing etc.

4) Should we drop all the bindings for bridging between such subsystems and do it all from userspace?

I think we may end up with a hybrid of the two, but need to be able to make it work in 'standard' cases without userspace being involved. That hybrid solution may well be devicetree overlay based... Unclear so far.
+ plenty of crazy things that 'might' make sense where there isn't even the pretence of representing real hardware.

Quoted 6 lines

5) Complex device interaction usecases. At the moment the ones I've come across are mostly contained in IIO. The moment we start sticking in MUXes, AFEs (Analog Front Ends) and straightforward analog sensors in the mix it can get fiddly. Swapping war stories may well be worthwhile on this. This stuff also turns up in ASoC for example so probably lessons to be learned from there.

V4L has a long history of this: if you have multiple video scalers in your pipeline, how do you control them? E.g. there can be scalers in the video sensor and in the SoC. We came up with the media controller device to expose the topology of the hardware and to be able to control each bit directly from userspace.

While the media controller is quite powerful and can be used to expose connections across subsystems, it also requires more work from userspace, and that's an area where not enough attention has been given.

Quoted 2 lines

The analog devices software defined radios are another possible case study.

Note that the V4L2 subsystem has support for several SDR devices. Probably another area where drivers can end up in different subsystems.

Quoted 10 lines

There of course may well be lessons to be learned from similar interactions elsewhere in the kernel.

There is a lot of history in how we ended up where we are (it all made absolute sense at the time). Sitting back and taking the time to discuss the future would be great. Whilst this might be solvable by email we've made no definitive progress for years
(and what has been made has been on a case by case basis deep in driver reviews.)

Quoted 4 lines

I threw comedi in the list above but, at the moment, I think the more likely direction there is a single userspace library abstracting the underlying subsystem (Analog Devices are working in that direction - perhaps Lars can offer more on that?).

When you get complex systems you probably have to move in that direction. I'd be interesting in such work going on in other subsystems. As mentioned above, the media controller needs something like that and it would be good to learn from others.

Quoted 8 lines

GPIO is another interesting case - a lot of hardware is capable of parallel sampling, some at high speeds. It's another area that is probably too specialist for this discussion, but if people want to dive into the details it might be interesting.

I think we have only a small amount of fuzz around the v4l boundary, but wanted to leave the door open if anyone wants to discuss that one further as it's come up a few times over recent years.

There is more of that than we wish, but quite often it is just dropped in the end as being too much work to solve and not quite important enough to spend the time and money on.

Quoted 18 lines

The SoC world is a major case of one device, many uses. Some SoCs are turning up with multiple ADC units, sometimes with different designs, sometimes simply so that the same hardware can be used for different things.

A few suggestions for people:

Guenter Roeck (hmwon)
Dmitry Torokhov (input)
Me (Jonathan Cameron - IIO)

Someone thermal related (not sure who would be most interested) Someone power supply related? (Guenter, any suggestions?)

Someone device tree related (Mark Rutland? Rob Herring?)

Linus Waleij (gpio)

Quoted 8 lines
Quoted 6 lines

Mark Brown - as some of concepts of IIO bridging and how far it can be taken (everything via a bridge driver) came from discussions with him on SoC ADC handling.

Lar-Peter Clausen for generally doing insane things with ADCs + having one foot in the userspace side of things.

Laurent Pinchart or myself for v4l/mc

I'd love to *not* take part in the discussion, but that would imply living in a world where the problem doesn't exist, and I don't know of any :-) As we have to solve it, I'm interested in this topic.

Signature · 3 lines
Regards,

Laurent Pinchart
#4 lore raw
In reply to Jonathan Cameron
Quoted 2 lines · Jonathan Cameron

4) Should we drop all the bindings for bridging between such subsystems and do it all from userspace?

This clearly doesn't seem sensible, we have plenty of hardware that's obviously wired up in a fixed function format even if it could be general purpose and several duplicated ABIs (eg, hwmon and power_supply have clear overlap).

Quoted 5 lines

I think we may end up with a hybrid of the two, but need to be able to make it work in 'standard' cases without userspace being involved. That hybrid solution may well be devicetree overlay based... Unclear so far.
+ plenty of crazy things that 'might' make sense where there isn't even the pretence of representing real hardware.

I'm definitely seeing people choosing their kernel subsystem on the basis of libiio.

#5 lore raw
In reply to Jonathan Cameron
Quoted 38 lines · Jonathan Cameron

Hi All,

This topic would be around the way the various subsystems interact, in the rough area of 'sensors' (I haven't yet had much of an issue with subsystem crossing with output devices but maybe that's just over the next hill!)

Scope may well be wider but includes:
* input (some of it)
* hwmon
* iio
* comedi(?)
* thermal
* power/battery
* gpio - the blur between gpios and beaglescope / PLC type I/O.
* v4l - when does a device jump from being a multi pixel thermopile to a thermal camera? Smart fingerprint scanners etc. Gesture recognition sensors (effectively 9ish pixel cameras)
* Lots of random things we haven't seen yet.

All these make use of devices that are at their hearts ADCs. Whilst it would nice if the underlying devices always fell into one clear category that is not always the case. Hardware descriptions (e.g. device tree) are an important area of contention.

The examples that follow are mostly based around IIO interactions with hwmon / input as that's my area of expertise. However, I know that similar issues occur at other boundaries.

Guenter Roeck has been working on hwmon / thermal interfacing recently. We also regularly push drivers one way or the other around that boundary.

Dmitry and I often have to decide on the IIO / input scope boundary (and we've moved it over time) + a number of cross over drivers have turned up recently.

The other subsystems I encounter have been 'quieter' in their interactions with me but many of the issues below apply to them as well.

Quoted 7 lines

A rough list of related topics that might be worth face to face discussion includes:

1) Are we happy with the somewhat adhoc divisions of subsystems as they stand?

I am personally happy enough with this, but it's worth checking

Same here.

Quoted 4 lines

everyone else is. This probably causes more pain for new submitters than it does for old hands. Perhaps we can come up with a checklist style doc to save people having to ask.

I think that would be very useful.

Quoted 2 lines

We have had quite a few cases where a whole driver is submitted to the 'wrong' subsystem. Not the best first interaction with the kernel

... especially to avoid that situation.

Quoted 24 lines

world for new contributors. (This sort of area is attracts newbies as it's relatively simple and the hardware is fairly cheap and more than once their first experience with review as exactly this!)

If we aren't happy, what do we do about it?

I'm hoping we are happy or, at least, resigned to the current state of affairs, but think the question should be asked from time to time as the answer may well change + the discussion will highlight pinch points we can work on.

2) Bridging drivers - there will always be cross over cases. E.g. Generic ADCs used for battery voltage monitoring. We have a number of bridges in mainline already and others out of tree. How important are these and what features would make them more useful?

Is it ever sensible to have two drivers for the same part because the use cases are so different? (I hope there is always a better way but maybe not)

This covers both generic bridges (e.g. iio-hwmon) and device specific bridges (a good example of a touch screen driver turned up in my inbox yesterday).

Quoted 7 lines

3) Bindings (device tree and similar). When it is appropriate to use a device tree to describe the overlap (bridging of channels)? Sometimes there is obvious real hardware involved (a thermistor on a generic ADC input for example), but there are many grey areas.

Finally bashing out an answer to the issue device tree maintainers have with the iio-hwmon bindings would be great.

Agreed, it would be great if we can come up with a fix for that.

Quoted 6 lines

On a less biased note, that example is pretty fiddly to define but would act as a good basis to work from more generally.

We know we got it wrong (back in the dark ages), but it isn't obvious how to get it right!

Quoted 8 lines

IIO-input has been partly held outside mainline for years by this issue.

Also some practical design decisions to make around how to implement these mappings to work best with deferred probing etc.

4) Should we drop all the bindings for bridging between such subsystems and do it all from userspace?

I think that would be a terrible idea. My hope is that we should be able to present a system to the user as it is intended to be, and that should include the use case for given hardware. An ADC is not just an ADC, it has a use case. The use case may be obvious if it is used as input device, but if it is used to measure a voltage (or current, or temperature) it should be possible to express that as well, without having to have userspace involved.

Quoted 6 lines

I think we may end up with a hybrid of the two, but need to be able to make it work in 'standard' cases without userspace being involved. That hybrid solution may well be devicetree overlay based... Unclear so far.
+ plenty of crazy things that 'might' make sense where there isn't even the pretence of representing real hardware.

Not really sure if/how that would require overlays as long as the use case is static. If an ADC is used to measure a voltage on a given system, it is unlikely to be used for something else in the same system.

Quoted 38 lines

5) Complex device interaction usecases. At the moment the ones I've come across are mostly contained in IIO. The moment we start sticking in MUXes, AFEs (Analog Front Ends) and straightforward analog sensors in the mix it can get fiddly. Swapping war stories may well be worthwhile on this. This stuff also turns up in ASoC for example so probably lessons to be learned from there.

The analog devices software defined radios are another possible case study.

There of course may well be lessons to be learned from similar interactions elsewhere in the kernel.

There is a lot of history in how we ended up where we are (it all made absolute sense at the time). Sitting back and taking the time to discuss the future would be great. Whilst this might be solvable by email we've made no definitive progress for years
(and what has been made has been on a case by case basis deep in driver reviews.)

I threw comedi in the list above but, at the moment, I think the more likely direction there is a single userspace library abstracting the underlying subsystem (Analog Devices are working in that direction - perhaps Lars can offer more on that?).

GPIO is another interesting case - a lot of hardware is capable of parallel sampling, some at high speeds. It's another area that is probably too specialist for this discussion, but if people want to dive into the details it might be interesting.

I think we have only a small amount of fuzz around the v4l boundary, but wanted to leave the door open if anyone wants to discuss that one further as it's come up a few times over recent years.

The SoC world is a major case of one device, many uses. Some SoCs
are turning up with multiple ADC units, sometimes with different designs, sometimes simply so that the same hardware can be used for different things.

Quoted 8 lines

A few suggestions for people:

Guenter Roeck (hmwon)
Dmitry Torokhov (input)
Me (Jonathan Cameron - IIO)

Someone thermal related (not sure who would be most interested) Someone power supply related? (Guenter, any suggestions?)

Not really, other than the power subsystem maintainers. Question here might be if it would make sense to register battery charger drivers as hardware monitoring devices - with the new hwmon API that should be quite straightforward.

I keep thinking about power, but regulators are probably at least as relevant when it comes to interaction with hardware monitoring. Many PMBus chips have both hardware monitoring and voltage/current regulator functionality. The PMBus core implements some basic functionality along that line, and can register a chip as regulator if requested. Would it make sense to register regulators as hardware monitoring devices ? I don't really know, and never thought about it until now. Maybe Mark Brown and/or Liam Girdwood have some thoughts on this.

Thanks,
Guenter

Quoted 17 lines

Someone device tree related (Mark Rutland? Rob Herring?)

Linus Waleij (gpio)

Mark Brown - as some of concepts of IIO bridging and how far it can be taken (everything via a bridge driver) came from discussions with him on SoC ADC handling.

Lar-Peter Clausen for generally doing insane things with ADCs + having one foot in the userspace side of things.

Thanks,

Jonathan Cameron
IIO Maintainer

#6 lore raw
Quoted 108 lines · Guenter Roeck
Quoted 38 lines · Jonathan Cameron

Hi All,

This topic would be around the way the various subsystems interact, in the rough area of 'sensors' (I haven't yet had much of an issue with subsystem crossing with output devices but maybe that's just over the next hill!)

Scope may well be wider but includes:
* input (some of it)
* hwmon
* iio
* comedi(?)
* thermal
* power/battery
* gpio - the blur between gpios and beaglescope / PLC type I/O.
* v4l - when does a device jump from being a multi pixel thermopile to a thermal camera? Smart fingerprint scanners etc. Gesture recognition sensors (effectively 9ish pixel cameras)
* Lots of random things we haven't seen yet.

All these make use of devices that are at their hearts ADCs. Whilst it would nice if the underlying devices always fell into one clear category that is not always the case. Hardware descriptions (e.g. device tree) are an important area of contention.

The examples that follow are mostly based around IIO interactions with hwmon / input as that's my area of expertise. However, I know that similar issues occur at other boundaries.

Guenter Roeck has been working on hwmon / thermal interfacing recently. We also regularly push drivers one way or the other around that boundary.

Dmitry and I often have to decide on the IIO / input scope boundary (and we've moved it over time) + a number of cross over drivers have turned up recently.

The other subsystems I encounter have been 'quieter' in their interactions with me but many of the issues below apply to them as well.

Quoted 7 lines

A rough list of related topics that might be worth face to face discussion includes:

1) Are we happy with the somewhat adhoc divisions of subsystems as they stand?

I am personally happy enough with this, but it's worth checking

Same here.

Quoted 4 lines

everyone else is. This probably causes more pain for new submitters than it does for old hands. Perhaps we can come up with a checklist style doc to save people having to ask.

I think that would be very useful.

Quoted 2 lines

We have had quite a few cases where a whole driver is submitted to the 'wrong' subsystem. Not the best first interaction with the kernel

... especially to avoid that situation.

Quoted 24 lines

world for new contributors. (This sort of area is attracts newbies as it's relatively simple and the hardware is fairly cheap and more than once their first experience with review as exactly this!)

If we aren't happy, what do we do about it?

I'm hoping we are happy or, at least, resigned to the current state of affairs, but think the question should be asked from time to time as the answer may well change + the discussion will highlight pinch points we can work on.

2) Bridging drivers - there will always be cross over cases. E.g. Generic ADCs used for battery voltage monitoring. We have a number of bridges in mainline already and others out of tree. How important are these and what features would make them more useful?

Is it ever sensible to have two drivers for the same part because the use cases are so different? (I hope there is always a better way but maybe not)

This covers both generic bridges (e.g. iio-hwmon) and device specific bridges (a good example of a touch screen driver turned up in my inbox yesterday).

Quoted 7 lines

3) Bindings (device tree and similar). When it is appropriate to use a device tree to describe the overlap (bridging of channels)? Sometimes there is obvious real hardware involved (a thermistor on a generic ADC input for example), but there are many grey areas.

Finally bashing out an answer to the issue device tree maintainers have with the iio-hwmon bindings would be great.

Agreed, it would be great if we can come up with a fix for that.

Quoted 6 lines

On a less biased note, that example is pretty fiddly to define but would act as a good basis to work from more generally.

We know we got it wrong (back in the dark ages), but it isn't obvious how to get it right!

Quoted 8 lines

IIO-input has been partly held outside mainline for years by this issue.

Also some practical design decisions to make around how to implement these mappings to work best with deferred probing etc.

4) Should we drop all the bindings for bridging between such subsystems and do it all from userspace?

Quoted 6 lines

I think that would be a terrible idea. My hope is that we should be able to present a system to the user as it is intended to be, and that should include the use case for given hardware. An ADC is not just an ADC, it has a use case. The use case may be obvious if it is used as input device, but if it is used to measure a voltage (or current, or temperature) it should be possible to express that as well, without having to have userspace involved.

Totally agree. We might want to allow reconfiguring from userspace, but having initial configuration conveyed through DT/ACPI/whatever according to the original system purpose makes sense to me.

Thanks.

Signature · 1 line
Dmitry
#7 lore raw
Quoted 116 lines · Torokhov
Quoted 108 lines · Guenter Roeck
Quoted 38 lines · Jonathan Cameron

Hi All,

This topic would be around the way the various subsystems interact, in the rough area of 'sensors' (I haven't yet had much of an issue with subsystem crossing with output devices but maybe that's just over the next hill!)

Scope may well be wider but includes:
* input (some of it)
* hwmon
* iio
* comedi(?)
* thermal
* power/battery
* gpio - the blur between gpios and beaglescope / PLC type I/O.
* v4l - when does a device jump from being a multi pixel thermopile to a thermal camera? Smart fingerprint scanners etc. Gesture recognition sensors (effectively 9ish pixel cameras)
* Lots of random things we haven't seen yet.

All these make use of devices that are at their hearts ADCs. Whilst it would nice if the underlying devices always fell into one clear category that is not always the case. Hardware descriptions (e.g. device tree) are an important area of contention.

The examples that follow are mostly based around IIO interactions with hwmon / input as that's my area of expertise. However, I know that similar issues occur at other boundaries.

Guenter Roeck has been working on hwmon / thermal interfacing recently. We also regularly push drivers one way or the other around that boundary.

Dmitry and I often have to decide on the IIO / input scope boundary (and we've moved it over time) + a number of cross over drivers have turned up recently.

The other subsystems I encounter have been 'quieter' in their interactions with me but many of the issues below apply to them as well.

Quoted 7 lines

A rough list of related topics that might be worth face to face discussion includes:

1) Are we happy with the somewhat adhoc divisions of subsystems as they stand?

I am personally happy enough with this, but it's worth checking

Same here.

Quoted 4 lines

everyone else is. This probably causes more pain for new submitters than it does for old hands. Perhaps we can come up with a checklist style doc to save people having to ask.

I think that would be very useful.

Quoted 2 lines

We have had quite a few cases where a whole driver is submitted to the 'wrong' subsystem. Not the best first interaction with the kernel

... especially to avoid that situation.

Quoted 24 lines

world for new contributors. (This sort of area is attracts newbies as it's relatively simple and the hardware is fairly cheap and more than once their first experience with review as exactly this!)

If we aren't happy, what do we do about it?

I'm hoping we are happy or, at least, resigned to the current state of affairs, but think the question should be asked from time to time as the answer may well change + the discussion will highlight pinch points we can work on.

2) Bridging drivers - there will always be cross over cases. E.g. Generic ADCs used for battery voltage monitoring. We have a number of bridges in mainline already and others out of tree. How important are these and what features would make them more useful?

Is it ever sensible to have two drivers for the same part because the use cases are so different? (I hope there is always a better way but maybe not)

This covers both generic bridges (e.g. iio-hwmon) and device specific bridges (a good example of a touch screen driver turned up in my inbox yesterday).

Quoted 7 lines

3) Bindings (device tree and similar). When it is appropriate to use a device tree to describe the overlap (bridging of channels)? Sometimes there is obvious real hardware involved (a thermistor on a generic ADC input for example), but there are many grey areas.

Finally bashing out an answer to the issue device tree maintainers have with the iio-hwmon bindings would be great.

Agreed, it would be great if we can come up with a fix for that.

Quoted 6 lines

On a less biased note, that example is pretty fiddly to define but would act as a good basis to work from more generally.

We know we got it wrong (back in the dark ages), but it isn't obvious how to get it right!

Quoted 8 lines

IIO-input has been partly held outside mainline for years by this issue.

Also some practical design decisions to make around how to implement these mappings to work best with deferred probing etc.

4) Should we drop all the bindings for bridging between such subsystems and do it all from userspace?

Quoted 6 lines

I think that would be a terrible idea. My hope is that we should be able to present a system to the user as it is intended to be, and that should include the use case for given hardware. An ADC is not just an ADC, it has a use case. The use case may be obvious if it is used as input device, but if it is used to measure a voltage (or current, or temperature) it should be possible to express that as well, without having to have userspace involved.

Quoted 3 lines

Totally agree. We might want to allow reconfiguring from userspace, but having initial configuration conveyed through DT/ACPI/whatever according to the original system purpose makes sense to me.

Agreed. There may be exceptions, but in general we need bindings or similar. The trick is how to make them both generic enough that they describe the hardware, and specific enough that we can easily use the bindings to make the relevant connections.

iio-hwmon always got grief for being Linux specific. Going as generic as this channel is a temperature sensor, this one is a power line etc makes for fiddly bindings as it's not clear what driver's problem they are. There is probably a middle ground we need to tread here.

There are plenty of single devices that have multiple uses as well some of which require very different handling. e.g. Accelerometer in a laptop. Does screen orientation, but may also handle parking a hard disk head during freefall, or even various augmented reality stuff (e.g. human input).

Jonathan

Quoted 3 lines

Thanks.

#8 lore raw
In reply to Guenter Roeck
Quoted 4 lines · Guenter Roeck

if requested. Would it make sense to register regulators as hardware monitoring devices ? I don't really know, and never thought about it until now. Maybe Mark Brown and/or Liam Girdwood have some thoughts on this.

It has been discussed before but it never seemed sensible to me. hwmon is a read only interface monitoring the actual state of the system while regulators are write only and if things are going wrong may not reflect the actual state of the system. Exporting write only information via hwmon seems to break expectations for users of it, but perhaps that's not what people are expecting from hwmon these days.

#9 lore raw
In reply to Guenter Roeck
Quoted 47 lines · Guenter Roeck
Quoted 38 lines · Jonathan Cameron

Hi All,

This topic would be around the way the various subsystems interact, in the rough area of 'sensors' (I haven't yet had much of an issue with subsystem crossing with output devices but maybe that's just over the next hill!)

Scope may well be wider but includes:
* input (some of it)
* hwmon
* iio
* comedi(?)
* thermal
* power/battery
* gpio - the blur between gpios and beaglescope / PLC type I/O.
* v4l - when does a device jump from being a multi pixel thermopile to a thermal camera? Smart fingerprint scanners etc. Gesture recognition sensors (effectively 9ish pixel cameras)
* Lots of random things we haven't seen yet.

All these make use of devices that are at their hearts ADCs. Whilst it would nice if the underlying devices always fell into one clear category that is not always the case. Hardware descriptions (e.g. device tree) are an important area of contention.

The examples that follow are mostly based around IIO interactions with hwmon / input as that's my area of expertise. However, I know that similar issues occur at other boundaries.

Guenter Roeck has been working on hwmon / thermal interfacing recently. We also regularly push drivers one way or the other around that boundary.

Dmitry and I often have to decide on the IIO / input scope boundary (and we've moved it over time) + a number of cross over drivers have turned up recently.

The other subsystems I encounter have been 'quieter' in their interactions with me but many of the issues below apply to them as well.

Quoted 7 lines

A rough list of related topics that might be worth face to face discussion includes:

1) Are we happy with the somewhat adhoc divisions of subsystems as they stand?

I am personally happy enough with this, but it's worth checking

Quoted 7 lines

Same here.

Quoted 4 lines

everyone else is. This probably causes more pain for new submitters than it does for old hands. Perhaps we can come up with a checklist style doc to save people having to ask.

I think that would be very useful.

This is one where we can perhaps bash out a rough version in advance to be talked over /refined in person.

Quoted 5 lines
Quoted 3 lines

We have had quite a few cases where a whole driver is submitted to the 'wrong' subsystem. Not the best first interaction with the kernel

... especially to avoid that situation.

Agreed. I feel guilty every time it happens....

Quoted 62 lines
Quoted 25 lines

world for new contributors. (This sort of area is attracts newbies as it's relatively simple and the hardware is fairly cheap and more than once their first experience with review as exactly this!)

If we aren't happy, what do we do about it?

I'm hoping we are happy or, at least, resigned to the current state of affairs, but think the question should be asked from time to time as the answer may well change + the discussion will highlight pinch points we can work on.

2) Bridging drivers - there will always be cross over cases. E.g. Generic ADCs used for battery voltage monitoring. We have a number of bridges in mainline already and others out of tree. How important are these and what features would make them more useful?

Is it ever sensible to have two drivers for the same part because the use cases are so different? (I hope there is always a better way but maybe not)

This covers both generic bridges (e.g. iio-hwmon) and device specific bridges (a good example of a touch screen driver turned up in my inbox yesterday).

Quoted 7 lines

3) Bindings (device tree and similar). When it is appropriate to use a device tree to describe the overlap (bridging of channels)? Sometimes there is obvious real hardware involved (a thermistor on a generic ADC input for example), but there are many grey areas.

Finally bashing out an answer to the issue device tree maintainers have with the iio-hwmon bindings would be great.

Agreed, it would be great if we can come up with a fix for that.

Quoted 6 lines

On a less biased note, that example is pretty fiddly to define but would act as a good basis to work from more generally.

We know we got it wrong (back in the dark ages), but it isn't obvious how to get it right!

Quoted 8 lines

IIO-input has been partly held outside mainline for years by this issue.

Also some practical design decisions to make around how to implement these mappings to work best with deferred probing etc.

4) Should we drop all the bindings for bridging between such subsystems and do it all from userspace?

I think that would be a terrible idea. My hope is that we should be able to present a system to the user as it is intended to be, and that should include the use case for given hardware. An ADC is not just an ADC, it has a use case. The use case may be obvious if it is used as input device, but if it is used to measure a voltage (or current, or temperature) it should be possible to express that as well, without having to have userspace involved.

Quoted 6 lines

I think we may end up with a hybrid of the two, but need to be able to make it work in 'standard' cases without userspace being involved. That hybrid solution may well be devicetree overlay based... Unclear so far.
+ plenty of crazy things that 'might' make sense where there isn't even the pretence of representing real hardware.

Quoted 3 lines

Not really sure if/how that would require overlays as long as the use case is static. If an ADC is used to measure a voltage on a given system, it is unlikely to be used for something else in the same system.

So you'd think ;) Lots of IIO usecases include ADCs measuring voltages on screw terminals and similar. A surprising number of industrial systems are build around generic boards, wired in weird and wonderful ways. They still want their 'hyrda' to work sensibly with standard kernels and present data in a way that makes sense. So probably a generic board with the mappings loaded via a device tree overlay. They just mess around with the overlay.

Quoted 45 lines
Quoted 45 lines

5) Complex device interaction usecases. At the moment the ones I've come across are mostly contained in IIO. The moment we start sticking in MUXes, AFEs (Analog Front Ends) and straightforward analog sensors in the mix it can get fiddly. Swapping war stories may well be worthwhile on this. This stuff also turns up in ASoC for example so probably lessons to be learned from there.

The analog devices software defined radios are another possible case study.

There of course may well be lessons to be learned from similar interactions elsewhere in the kernel.

There is a lot of history in how we ended up where we are (it all made absolute sense at the time). Sitting back and taking the time to discuss the future would be great. Whilst this might be solvable by email we've made no definitive progress for years
(and what has been made has been on a case by case basis deep in driver reviews.)

I threw comedi in the list above but, at the moment, I think the more likely direction there is a single userspace library abstracting the underlying subsystem (Analog Devices are working in that direction - perhaps Lars can offer more on that?).

GPIO is another interesting case - a lot of hardware is capable of parallel sampling, some at high speeds. It's another area that is probably too specialist for this discussion, but if people want to dive into the details it might be interesting.

I think we have only a small amount of fuzz around the v4l boundary, but wanted to leave the door open if anyone wants to discuss that one further as it's come up a few times over recent years.

The SoC world is a major case of one device, many uses. Some SoCs
are turning up with multiple ADC units, sometimes with different designs, sometimes simply so that the same hardware can be used for different things.

A few suggestions for people:

Guenter Roeck (hmwon)
Dmitry Torokhov (input)
Me (Jonathan Cameron - IIO)

Quoted 7 lines
Quoted 2 lines

Someone thermal related (not sure who would be most interested) Someone power supply related? (Guenter, any suggestions?)

Not really, other than the power subsystem maintainers. Question here might be if it would make sense to register battery charger drivers as hardware monitoring devices - with the new hwmon API that should be quite straightforward.

Anyone want to nominate themselves from that area?

Quoted 32 lines

I keep thinking about power, but regulators are probably at least as relevant when it comes to interaction with hardware monitoring. Many PMBus chips have both hardware monitoring and voltage/current regulator functionality. The PMBus core implements some basic functionality along that line, and can register a chip as regulator if requested. Would it make sense to register regulators as hardware monitoring devices ? I don't really know, and never thought about it until now. Maybe Mark Brown and/or Liam Girdwood have some thoughts on this.

Thanks,
Guenter

Quoted 17 lines

Someone device tree related (Mark Rutland? Rob Herring?)

Linus Waleij (gpio)

Mark Brown - as some of concepts of IIO bridging and how far it can be taken (everything via a bridge driver) came from discussions with him on SoC ADC handling.

Lar-Peter Clausen for generally doing insane things with ADCs + having one foot in the userspace side of things.

Thanks,

Jonathan Cameron
IIO Maintainer

#10 lore raw

Hi,

Quoted 6 lines · Jonathan Cameron
Quoted 5 lines
Quoted 1 line

Someone power supply related? (Guenter, any suggestions?)

Not really, other than the power subsystem maintainers. Question here might be if it would make sense to register battery charger drivers as hardware monitoring devices - with the new hwmon API that should be quite straightforward.

Anyone want to nominate themselves from that area?

Sorry, travelling to USA is not something I can financially afford until I have found an employer. Since [0] I basically maintained the power supply subsystem alone, so Dmitry and David are probably not the best candidates either.

If he's there, Krzysztof Kozlowski (added to CC) may be a good candidate, though.

[0] https://lkml.org/lkml/2014/7/16/621

-- Sebastian

#11 lore raw
In reply to Guenter Roeck

On 07/22/2016 05:29 AM, Guenter Roeck wrote:
[...]

Quoted 4 lines
Quoted 4 lines

4) Should we drop all the bindings for bridging between such subsystems and do it all from userspace?

Quoted 9 lines

I think that would be a terrible idea. My hope is that we should be able to present a system to the user as it is intended to be, and that should include
the use case for given hardware. An ADC is not just an ADC, it has a use case.
The use case may be obvious if it is used as input device, but if it is used to measure a voltage (or current, or temperature) it should be possible
to express that as well, without having to have userspace involved.

We need to differentiate between two different things. One is figuring out the function. If you have a general purpose ADC that is hooked up to some other hardware construct that usually gives you the function. E.g. a GP ADC connected to a resistor ladder with push-buttons is a input device. The same ADC connected to a thermocouple creates a temperature sensor. This can easily be implemented by describing the additional hardware properly in the DT and having a construct in the DT that establishes that this ADC is connected to this other thing. We can then use the in-kernel consumer API to get the raw ADC readings and translate them to a application specific result.

This is all OK and I don't think anybody has any problem with this. It is well within the scope of what devicetree is meant for and it has no operating system specific bits per se.

The other thing is choosing the userspace interface through which the data is exposed when the application falls in an area where it could be exposed through one of multiple APIs. E.g. a general purpose temperature sensor could be either IIO or hwmon. A accelerometer could either be input or IIO. Some user might prefer one interface over the other because they have existing application code that has support for this interface. Another user wants to use another interface because they have code for this interface.

This is the kind of bridge setup I'd rather very much avoid putting in the DT. Strictly speaking at the DT level you don't know what kind of application the users are going to run so you can't make that decission at that level. Sure if you build a very tightly coupled embedded application you get kernel, DT and userspace out of one hand and you know at the DT level which interface your application will require, but this is still a layering violation and should be avoided if possible.

Saying you want a IIO or hwmon interface in the DT is also very Linux specific. Another reason why we should avoid this.

In my opinion we should make the default interface what we think to the best of our knowledge is the appropriate interface for this type of device and not offer an option to overwrite this through the DT. Where things are on the edge we might let the driver author decide since they might have a valid usecase for that interface.

In addition to that I think we should offer the possibility to instantiate bridges from userspace to handle the case where somebody really wants to use the other interface. Theses bridges should solely be about changing the delivery method and not about changing the data type.

E.g. a userspace bridge should not be used to create a input device for a resistor ladder connected to a GP ADC as this would change the meaning of the data. On the other hand creating a hwmon to IIO bridge for a temperature sensor should be OK since both report the same temperature value.

I'm not sure what I think of this yet, but for completeness I want to mention that we could also get a similar result by implementing the bridge in userspace by LD_PRELOAD'ing. Maybe this is where is belongs?

Another thing that is up for discussion is what should do with the ADC device in case there is additional hardware connected. Should we still expose the userspace interface to raw ADC device? If not how would we figure out that we shouldn't considering that the consumer is probed after the provider.

- Lars

#12 lore raw
Quoted 62 lines · Lars-Peter Clausen

On 07/22/2016 05:29 AM, Guenter Roeck wrote:
[...]

Quoted 4 lines
Quoted 4 lines

4) Should we drop all the bindings for bridging between such subsystems and do it all from userspace?

Quoted 9 lines

I think that would be a terrible idea. My hope is that we should be able to present a system to the user as it is intended to be, and that should include
the use case for given hardware. An ADC is not just an ADC, it has a use case.
The use case may be obvious if it is used as input device, but if it is used to measure a voltage (or current, or temperature) it should be possible
to express that as well, without having to have userspace involved.

We need to differentiate between two different things. One is figuring out the function. If you have a general purpose ADC that is hooked up to some other hardware construct that usually gives you the function. E.g. a GP ADC connected to a resistor ladder with push-buttons is a input device. The same ADC connected to a thermocouple creates a temperature sensor. This can easily be implemented by describing the additional hardware properly in the DT and having a construct in the DT that establishes that this ADC is connected to this other thing. We can then use the in-kernel consumer API to get the raw ADC readings and translate them to a application specific result.

This is all OK and I don't think anybody has any problem with this. It is well within the scope of what devicetree is meant for and it has no operating system specific bits per se.

The other thing is choosing the userspace interface through which the data is exposed when the application falls in an area where it could be exposed through one of multiple APIs. E.g. a general purpose temperature sensor could be either IIO or hwmon. A accelerometer could either be input or IIO. Some user might prefer one interface over the other because they have existing application code that has support for this interface. Another user wants to use another interface because they have code for this interface.

This is the kind of bridge setup I'd rather very much avoid putting in the DT. Strictly speaking at the DT level you don't know what kind of application the users are going to run so you can't make that decission at that level. Sure if you build a very tightly coupled embedded application you get kernel, DT and userspace out of one hand and you know at the DT level which interface your application will require, but this is still a layering violation and should be avoided if possible.

Saying you want a IIO or hwmon interface in the DT is also very Linux specific. Another reason why we should avoid this.

In my opinion we should make the default interface what we think to the best of our knowledge is the appropriate interface for this type of device and not offer an option to overwrite this through the DT. Where things are on the edge we might let the driver author decide since they might have a valid usecase for that interface.

In addition to that I think we should offer the possibility to instantiate bridges from userspace to handle the case where somebody really wants to use the other interface. Theses bridges should solely be about changing the delivery method and not about changing the data type.

Quoted 5 lines

E.g. a userspace bridge should not be used to create a input device for a resistor ladder connected to a GP ADC as this would change the meaning of the data. On the other hand creating a hwmon to IIO bridge for a temperature sensor should be OK since both report the same temperature value.

That's the sort of balance I think we need to reach as well.

Quoted 10 lines

I'm not sure what I think of this yet, but for completeness I want to mention that we could also get a similar result by implementing the bridge in userspace by LD_PRELOAD'ing. Maybe this is where is belongs?

Another thing that is up for discussion is what should do with the ADC device in case there is additional hardware connected. Should we still expose the userspace interface to raw ADC device? If not how would we figure out that we shouldn't considering that the consumer is probed after the provider.

Is there any harm in doing so?

There was an effort a few years back to separate the backend and front end of IIO explicitly to allow the dropping of the userspace interface. In the extreme case could we 'solve' the exposing the wrong thing case by requiring every channel to be mapped to a client? So if you have a general purpose channel, it has to be mapped as such - interfaces are effectively only provided on demand. Once an interface has been provided in the extreme case we could lock out any more from using that channel.

Doing this for all subsystems with fuzzy edge cases would be a pain however!

Jonathan

Quoted 3 lines

- Lars

#13 lore raw
In reply to Lars-Peter Clausen
Quoted 48 lines · Lars-Peter Clausen

On 07/22/2016 05:29 AM, Guenter Roeck wrote:
[...]

Quoted 4 lines
Quoted 4 lines

4) Should we drop all the bindings for bridging between such subsystems and do it all from userspace?

Quoted 9 lines

I think that would be a terrible idea. My hope is that we should be able to present a system to the user as it is intended to be, and that should include
the use case for given hardware. An ADC is not just an ADC, it has a use case.
The use case may be obvious if it is used as input device, but if it is used to measure a voltage (or current, or temperature) it should be possible
to express that as well, without having to have userspace involved.

We need to differentiate between two different things. One is figuring out the function. If you have a general purpose ADC that is hooked up to some other hardware construct that usually gives you the function. E.g. a GP ADC connected to a resistor ladder with push-buttons is a input device. The same ADC connected to a thermocouple creates a temperature sensor. This can easily be implemented by describing the additional hardware properly in the DT and having a construct in the DT that establishes that this ADC is connected to this other thing. We can then use the in-kernel consumer API to get the raw ADC readings and translate them to a application specific result.

This is all OK and I don't think anybody has any problem with this. It is well within the scope of what devicetree is meant for and it has no operating system specific bits per se.

The other thing is choosing the userspace interface through which the data is exposed when the application falls in an area where it could be exposed through one of multiple APIs. E.g. a general purpose temperature sensor could be either IIO or hwmon. A accelerometer could either be input or IIO. Some user might prefer one interface over the other because they have existing application code that has support for this interface. Another user wants to use another interface because they have code for this interface.

This is the kind of bridge setup I'd rather very much avoid putting in the DT. Strictly speaking at the DT level you don't know what kind of application the users are going to run so you can't make that decission at that level. Sure if you build a very tightly coupled embedded application you get kernel, DT and userspace out of one hand and you know at the DT level which interface your application will require, but this is still a layering violation and should be avoided if possible.

Quoted 3 lines

Saying you want a IIO or hwmon interface in the DT is also very Linux specific. Another reason why we should avoid this.

That ambiguity has not been the case at least in the use cases I was involved in. It just happened that some hardware engineers decided that they wanted to use a common ADC to measure voltages on a board (rather, on all the boards built by that company). So the ADCs are connected to board voltages, and the intent was very clear - to be used for hardware monitoring.

I am sure it can be expressed that an ADC is connected to, say, a regulator output. The question is how to express the intent to use that connection for the purpose of hardware monitoring. I think it should be possible for the board designer to make a statement about the intended use of the hardware.

Guenter

Quoted 30 lines

In my opinion we should make the default interface what we think to the best of our knowledge is the appropriate interface for this type of device and not offer an option to overwrite this through the DT. Where things are on the edge we might let the driver author decide since they might have a valid usecase for that interface.

In addition to that I think we should offer the possibility to instantiate bridges from userspace to handle the case where somebody really wants to use the other interface. Theses bridges should solely be about changing the delivery method and not about changing the data type.

E.g. a userspace bridge should not be used to create a input device for a resistor ladder connected to a GP ADC as this would change the meaning of the data. On the other hand creating a hwmon to IIO bridge for a temperature sensor should be OK since both report the same temperature value.

I'm not sure what I think of this yet, but for completeness I want to mention that we could also get a similar result by implementing the bridge in userspace by LD_PRELOAD'ing. Maybe this is where is belongs?

Another thing that is up for discussion is what should do with the ADC device in case there is additional hardware connected. Should we still expose the userspace interface to raw ADC device? If not how would we figure out that we shouldn't considering that the consumer is probed after the provider.

- Lars

#14 lore raw
Quoted 80 lines · Guenter Roeck
Quoted 4 lines · Lars-Peter Clausen

On 07/22/2016 05:29 AM, Guenter Roeck wrote:
[...]

Quoted 2 lines
Quoted 2 lines

4) Should we drop all the bindings for bridging between such

subsystems

Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

and do it all from userspace?

I think that would be a terrible idea. My hope is that we should be

able

Quoted 1 line
Quoted 1 line

to present a system to the user as it is intended to be, and that

should

Quoted 2 lines
Quoted 2 lines

include
the use case for given hardware. An ADC is not just an ADC, it has a

use

Quoted 2 lines
Quoted 2 lines

case.
The use case may be obvious if it is used as input device, but if it

is

Quoted 5 lines
Quoted 3 lines

used to measure a voltage (or current, or temperature) it should be possible
to express that as well, without having to have userspace involved.

We need to differentiate between two different things. One is

figuring

Quoted 1 line

out the function. If you have a general purpose ADC that is hooked up

to

Quoted 1 line

some other hardware construct that usually gives you the function.

E.g.

Quoted 2 lines

a GP ADC connected to a resistor ladder with push-buttons is a input device. The same ADC connected to a thermocouple creates a

temperature

Quoted 3 lines

sensor. This can easily be implemented by describing the additional hardware properly in the DT and having a construct in the DT that establishes that this ADC is connected to this other thing. We can

then

Quoted 1 line

use the in-kernel consumer API to get the raw ADC readings and

translate

Quoted 3 lines

them to a application specific result.

This is all OK and I don't think anybody has any problem with this.

It

Quoted 1 line

is well within the scope of what devicetree is meant for and it has

no

Quoted 4 lines

operating system specific bits per se.

The other thing is choosing the userspace interface through which the data is exposed when the application falls in an area where it could

be

Quoted 1 line

exposed through one of multiple APIs. E.g. a general purpose

temperature

Quoted 4 lines

sensor could be either IIO or hwmon. A accelerometer could either be input or IIO. Some user might prefer one interface over the other because they have existing application code that has support for this interface. Another user wants to use another interface because they

have

Quoted 3 lines

code for this interface.

This is the kind of bridge setup I'd rather very much avoid putting

in

Quoted 2 lines

the DT. Strictly speaking at the DT level you don't know what kind of application the users are going to run so you can't make that

decission

Quoted 3 lines

at that level. Sure if you build a very tightly coupled embedded application you get kernel, DT and userspace out of one hand and you know at the DT level which interface your application will require,

but

Quoted 5 lines

this is still a layering violation and should be avoided if possible.

Saying you want a IIO or hwmon interface in the DT is also very Linux specific. Another reason why we should avoid this.

That ambiguity has not been the case at least in the use cases I was involved in. It just happened that some hardware engineers decided that they wanted to use a common ADC to measure voltages on a board (rather, on all the boards built by that company). So the ADCs are connected to board
voltages, and the intent was very clear - to be used for hardware monitoring.

Quoted 7 lines

I am sure it can be expressed that an ADC is connected to, say, a regulator
output. The question is how to express the intent to use that connection
for the purpose of hardware monitoring. I think it should be possible for the board designer to make a statement about the intended use of the hardware.

Arguably, this is another case of describing wiring. If it's wired to monitor something like a power line, it's that we should try to figure out how to describe.

Maybe your suggestion of making it linked to a regulator is one way of doing it. 'This adc channel is measuring this regulated channel'. Always a hwmon case
to my mind. Going to end up with sets of ad-hoc looking rules though to make that
work if we can do it at all.

This was the device tree maintainer suggestion IIRC on how we should have done the IIO-HWMON bindings..

Jonathan

Quoted 44 lines

Guenter

Quoted 1 line

In my opinion we should make the default interface what we think to

the

Quoted 2 lines

best of our knowledge is the appropriate interface for this type of device and not offer an option to overwrite this through the DT.

Where

Quoted 1 line

things are on the edge we might let the driver author decide since

they

Quoted 5 lines

might have a valid usecase for that interface.

In addition to that I think we should offer the possibility to instantiate bridges from userspace to handle the case where somebody really wants to use the other interface. Theses bridges should solely

be

Quoted 1 line

about changing the delivery method and not about changing the data

type.

Quoted 2 lines

E.g. a userspace bridge should not be used to create a input device

for

Quoted 1 line

a resistor ladder connected to a GP ADC as this would change the

meaning

Quoted 2 lines

of the data. On the other hand creating a hwmon to IIO bridge for a temperature sensor should be OK since both report the same

temperature

Quoted 5 lines

value.

I'm not sure what I think of this yet, but for completeness I want to mention that we could also get a similar result by implementing the bridge in userspace by LD_PRELOAD'ing. Maybe this is where is

belongs?

Quoted 2 lines

Another thing that is up for discussion is what should do with the

ADC

Quoted 1 line

device in case there is additional hardware connected. Should we

still

Quoted 7 lines

expose the userspace interface to raw ADC device? If not how would we figure out that we shouldn't considering that the consumer is probed after the provider.

- Lars

Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.
#15 lore raw
In reply to Jonathan Cameron
Quoted 4 lines · Jonathan Cameron

This topic would be around the way the various subsystems interact, in the rough area of 'sensors' (I haven't yet had much of an issue with subsystem crossing with output devices but maybe that's just over the next hill!)

Quoted 8 lines

Scope may well be wider but includes:
* input (some of it)
* hwmon
* iio
* comedi(?)
* thermal
* power/battery
* gpio - the blur between gpios and beaglescope / PLC type I/O.

I'm in. The new chardev ABI was designed to fit play/well with IIO, for example I added a 64bit timestamp also to GPIO events so that people could do what I guess they refer to as sensor fusion mixing and blending sensor input events with GPIO events.
(https://en.wikipedia.org/wiki/Sensor_fusion)

What I wanted to bring up ASAP was when I noticed in linux-next commit bc2b7dab629a51e8beb5fda4222c62a23b729f26 "iio:core: timestamping clock selection support"
that we should probably move this code to e.g. drivers/base or drivers/lib so I can reuse the same ABI in sysfs for GPIO event timestamping.

I think timestamping events uniformly is something that is important to all userspaces.

Quoted 3 lines

Dmitry and I often have to decide on the IIO / input scope boundary (and we've moved it over time) + a number of cross over drivers have turned up recently.

(snip from elsewhere)

Quoted 3 lines

Is it ever sensible to have two drivers for the same part because the use cases are so different? (I hope there is always a better way but maybe not)

I have one on my knee as I want to make a proper IIO driver for MPU-3050 gyroscope which has a (very simplified) driver in drivers/input/misc/mpu3050.c that was merged before the advent of the IIO subsystem.

For the new driver I have these options:

- Keep both drivers and consider them for different usecases. of the same hardware that require different Kconfig (seems very unelegant to me)

- Add some input event calls into the IIO driver so we can phase over the older uses to the new driver.

- Drop the input event handling since gyroscopes should not be handled by input anyway

- Develop a generic IIO-gyroscope-to-input event bridge. (Phew seems complex.)

(elsewhere)

Quoted 1 line

IIO-input has been partly held outside mainline for years by this issue.

Is there such a thing? gitweb?

Quoted 2 lines

1) Are we happy with the somewhat adhoc divisions of subsystems as they stand?

IETF has this motto "rough consensus and running code", and for me that is the working assumption to most things driver-subsystem-division-of-responsibilities-related.

Quoted 5 lines

We have had quite a few cases where a whole driver is submitted to the 'wrong' subsystem. Not the best first interaction with the kernel world for new contributors. (This sort of area is attracts newbies as it's relatively simple and the hardware is fairly cheap and more than once their first experience with review as exactly this!)

What is sometimes lacking is users who are clever enough to take the latest kernel (or preferably even -next) and git grep for a few strings related to the hardware they want to support, and be revealed the fact that there is already a driver for it.

If there isn't, their problem is that of finding the right subsystem to pigeon-hole their hardware into. It typically requires a kernel old grumpy person saying "no, THIS thing goes THERE, THAT thing goes THERE". Unfortunately I think this is pretty hard to codify.

Quoted 1 line

If we aren't happy, what do we do about it?

More helpful kernel generalists...

Quoted 4 lines

GPIO is another interesting case - a lot of hardware is capable of parallel sampling, some at high speeds. It's another area that is probably too specialist for this discussion, but if people want to dive into the details it might be interesting.

I currently have (for v4.8) an ABI which makes i possible to e.g. read 32 simultaneous lines with one userspace/kernelspace swap, if it ends up as a readl() from a register.

The userspace ABI has a limitation to 64 values, so it would be possible to have some driver read 64 GPIO lines and return them in a u8 vector back to userspace in one go.

For an oscilloscope or signal analyzer usecase with more than 64 channels needing to be sampled simultaneously, we may need to add another "large GPIO read" ABI, and also probably a new driver API interface to achieve it. I've never seen hardware that can read more values in parallel than 32 though, I chose 64 to get some headroom.

Quoted 1 line

Linus Waleij (gpio)

Sure, if travels etc work out and I get invited I'd come running.

Yours,
Linus Walleij

#16 lore raw
Quoted 15 lines · Linus Walleij
Quoted 4 lines · Jonathan Cameron

This topic would be around the way the various subsystems interact, in the rough area of 'sensors' (I haven't yet had much of an issue with subsystem crossing with output devices but maybe that's just over the next hill!)

Quoted 8 lines

Scope may well be wider but includes:
* input (some of it)
* hwmon
* iio
* comedi(?)
* thermal
* power/battery
* gpio - the blur between gpios and beaglescope / PLC type I/O.

Quoted 5 lines

I'm in. The new chardev ABI was designed to fit play/well with IIO, for example I added a 64bit timestamp also to GPIO events so that people could do what I guess they refer to as sensor fusion mixing and blending sensor input events with GPIO events.
(https://en.wikipedia.org/wiki/Sensor_fusion)

Great.

Quoted 10 lines

What I wanted to bring up ASAP was when I noticed in linux-next commit bc2b7dab629a51e8beb5fda4222c62a23b729f26 "iio:core: timestamping clock selection support"
that we should probably move this code to e.g. drivers/base or drivers/lib so I can reuse the same ABI in sysfs for GPIO event timestamping.

I think timestamping events uniformly is something that is important to all userspaces.

Seems sensible to me.
Should be easy enough to move IIO over to a shared implementation without ABI breakage.

Quoted 16 lines
Quoted 4 lines

Dmitry and I often have to decide on the IIO / input scope boundary (and we've moved it over time) + a number of cross over drivers have turned up recently.

(snip from elsewhere)

Quoted 3 lines

Is it ever sensible to have two drivers for the same part because the use cases are so different? (I hope there is always a better way but maybe not)

I have one on my knee as I want to make a proper IIO driver for MPU-3050 gyroscope which has a (very simplified) driver in drivers/input/misc/mpu3050.c that was merged before the advent of the IIO subsystem.

Quoted 5 lines

For the new driver I have these options:

- Keep both drivers and consider them for different usecases. of the same hardware that require different Kconfig (seems very unelegant to me)

Agreed.

Quoted 3 lines

- Add some input event calls into the IIO driver so we can phase over the older uses to the new driver.

I'd push back on this probably, though if the old driver has is in heavy use we'd have little choice about maintaining the ABI for a good long time. So whilst it would feel nasty, it might be the only way to keep an 'identical' ABI in place.

Quoted 3 lines

- Drop the input event handling since gyroscopes should not be handled by input anyway

Would break userspace unfortunately...

Quoted 3 lines

- Develop a generic IIO-gyroscope-to-input event bridge. (Phew seems complex.)

Not that hard actually - see the below. The infrastructure to do this stuff went in a long time back, but we are only just seeing mainline users showing up. Given we get bug fixes for that core code from time to time, presumably there are non mainline users. The core rewrite that led to that was very useful in making us tidy some corners up anyway.

Quoted 5 lines

(elsewhere)

Quoted 1 line

IIO-input has been partly held outside mainline for years by this issue.

Is there such a thing? gitweb?

Umm. Hides in old emails...
http://www.spinics.net/lists/linux-iio/msg06881.html
Hohum. Nearly 4 years since I revised that oops.

Quoted 25 lines
Quoted 3 lines

1) Are we happy with the somewhat adhoc divisions of subsystems as they stand?

IETF has this motto "rough consensus and running code", and for me that is the working assumption to most things driver-subsystem-division-of-responsibilities-related.

Quoted 5 lines

We have had quite a few cases where a whole driver is submitted to the 'wrong' subsystem. Not the best first interaction with the kernel world for new contributors. (This sort of area is attracts newbies as it's relatively simple and the hardware is fairly cheap and more than once their first experience with review as exactly this!)

What is sometimes lacking is users who are clever enough to take the latest kernel (or preferably even -next) and git grep for a few strings related to the hardware they want to support, and be revealed the fact that there is already a driver for it.

If there isn't, their problem is that of finding the right subsystem to pigeon-hole their hardware into. It typically requires a kernel old grumpy person saying "no, THIS thing goes THERE, THAT thing goes THERE". Unfortunately I think this is pretty hard to codify.

Quoted 3 lines
Quoted 1 line

If we aren't happy, what do we do about it?

More helpful kernel generalists...

:)

Quoted 9 lines
Quoted 5 lines

GPIO is another interesting case - a lot of hardware is capable of parallel sampling, some at high speeds. It's another area that is probably too specialist for this discussion, but if people want to dive into the details it might be interesting.

I currently have (for v4.8) an ABI which makes i possible to e.g. read 32 simultaneous lines with one userspace/kernelspace swap, if it ends up as a readl() from a register.

Cool. I've kind of lost track of where you were going with that.

Quoted 11 lines

The userspace ABI has a limitation to 64 values, so it would be possible to have some driver read 64 GPIO lines and return them in a u8 vector back to userspace in one go.

For an oscilloscope or signal analyzer usecase with more than 64 channels needing to be sampled simultaneously, we may need to add another "large GPIO read" ABI, and also probably a new driver API interface to achieve it. I've never seen hardware that can read more values in parallel than 32 though, I chose 64 to get some headroom.

Very nice.

That gets a long way for SoC gpio type cases, but there is also the class of external fpga or similar devices with their own buffering. This stuff fits well with the work Lars did on DMA buffers in IIO. The PRU stuff in beagle logic falls into the same category.

Also plenty of simple devices have a few inputs on them sampled in parallel with their 'ADC' channels, some of these have hardware fifos etc to make life more interesting. Reading isn't low latency, but it is high frequency.

Now neither of these are really GPIOs and probably shouldn't be treated as such. However, clearly there is a blurred line between the two. Particularly when we consider triggered polling of GPIOs. At somepoint I want to look at buffer fusion for IIO. Can do it now with a driver consuming multiple IIO buffers, but it would be ugly. (By this I merely mean fusing the data streams from multiple devices running of one sampling trigger into a single stream, not an clever data processing)

Quoted 4 lines
Quoted 2 lines

Linus Waleij (gpio)

Sure, if travels etc work out and I get invited I'd come running.

Fingers crossed then!

Jonathan

Quoted 4 lines

Yours,
Linus Walleij

#17 lore raw
In reply to Linus Walleij

Hi Linus,

Quoted 107 lines · On Friday 22 Jul 2016 14:04:07 Linus Walleij wrote:
Quoted 4 lines · Jonathan Cameron

This topic would be around the way the various subsystems interact, in the rough area of 'sensors' (I haven't yet had much of an issue with subsystem crossing with output devices but maybe that's just over the next hill!)

Quoted 8 lines

Scope may well be wider but includes:
* input (some of it)
* hwmon
* iio
* comedi(?)
* thermal
* power/battery
* gpio - the blur between gpios and beaglescope / PLC type I/O.

I'm in. The new chardev ABI was designed to fit play/well with IIO, for example I added a 64bit timestamp also to GPIO events so that people could do what I guess they refer to as sensor fusion mixing and blending sensor input events with GPIO events.
(https://en.wikipedia.org/wiki/Sensor_fusion)

What I wanted to bring up ASAP was when I noticed in linux-next commit bc2b7dab629a51e8beb5fda4222c62a23b729f26 "iio:core: timestamping clock selection support"
that we should probably move this code to e.g. drivers/base or drivers/lib so I can reuse the same ABI in sysfs for GPIO event timestamping.

I think timestamping events uniformly is something that is important to all userspaces.

Quoted 3 lines

Dmitry and I often have to decide on the IIO / input scope boundary (and we've moved it over time) + a number of cross over drivers have turned up recently.

(snip from elsewhere)

Quoted 3 lines

Is it ever sensible to have two drivers for the same part because the use cases are so different? (I hope there is always a better way but maybe not)

I have one on my knee as I want to make a proper IIO driver for MPU-3050 gyroscope which has a (very simplified) driver in drivers/input/misc/mpu3050.c that was merged before the advent of the IIO subsystem.

For the new driver I have these options:

- Keep both drivers and consider them for different usecases. of the same hardware that require different Kconfig (seems very unelegant to me)

- Add some input event calls into the IIO driver so we can phase over the older uses to the new driver.

- Drop the input event handling since gyroscopes should not be handled by input anyway

- Develop a generic IIO-gyroscope-to-input event bridge. (Phew seems complex.)

(elsewhere)

Quoted 1 line

IIO-input has been partly held outside mainline for years by this issue.

Is there such a thing? gitweb?

Quoted 2 lines

1) Are we happy with the somewhat adhoc divisions of subsystems as they stand?

IETF has this motto "rough consensus and running code", and for me that is the working assumption to most things driver-subsystem-division-of-responsibilities-related.

Quoted 5 lines

We have had quite a few cases where a whole driver is submitted to the 'wrong' subsystem. Not the best first interaction with the kernel world for new contributors. (This sort of area is attracts newbies as it's relatively simple and the hardware is fairly cheap and more than once their first experience with review as exactly this!)

What is sometimes lacking is users who are clever enough to take the latest kernel (or preferably even -next) and git grep for a few strings related to the hardware they want to support, and be revealed the fact that there is already a driver for it.

If there isn't, their problem is that of finding the right subsystem to pigeon-hole their hardware into. It typically requires a kernel old grumpy person saying "no, THIS thing goes THERE, THAT thing goes THERE". Unfortunately I think this is pretty hard to codify.

Quoted 1 line

If we aren't happy, what do we do about it?

More helpful kernel generalists...

Quoted 4 lines

GPIO is another interesting case - a lot of hardware is capable of parallel sampling, some at high speeds. It's another area that is probably too specialist for this discussion, but if people want to dive into the details it might be interesting.

I currently have (for v4.8) an ABI which makes i possible to e.g. read 32 simultaneous lines with one userspace/kernelspace swap, if it ends up as a readl() from a register.

The userspace ABI has a limitation to 64 values, so it would be possible to have some driver read 64 GPIO lines and return them in a u8 vector back to userspace in one go.

Quoted 6 lines

For an oscilloscope or signal analyzer usecase with more than 64 channels needing to be sampled simultaneously, we may need to add another "large GPIO read" ABI, and also probably a new driver API interface to achieve it. I've never seen hardware that can read more values in parallel than 32 though, I chose 64 to get some headroom.

Such a use case would put strong real-time constraints on the system, I wonder how useful a GPIO-based signal analyzer would be. Are you aware of any GPIO hardware that can be programmed to sample at a precise point in time (or perhaps even periodically) ?

Quoted 3 lines
Quoted 1 line

Linus Waleij (gpio)

Sure, if travels etc work out and I get invited I'd come running.

Signature · 3 lines
Regards,

Laurent Pinchart
#18 lore raw
Quoted 117 lines · Laurent Pinchart

Hi Linus,

Quoted 107 lines · On Friday 22 Jul 2016 14:04:07 Linus Walleij wrote:
Quoted 4 lines · Jonathan Cameron

This topic would be around the way the various subsystems interact, in the rough area of 'sensors' (I haven't yet had much of an issue with subsystem crossing with output devices but maybe that's just over the next hill!)

Quoted 8 lines

Scope may well be wider but includes:
* input (some of it)
* hwmon
* iio
* comedi(?)
* thermal
* power/battery
* gpio - the blur between gpios and beaglescope / PLC type I/O.

I'm in. The new chardev ABI was designed to fit play/well with IIO, for example I added a 64bit timestamp also to GPIO events so that people could do what I guess they refer to as sensor fusion mixing and blending sensor input events with GPIO events.
(https://en.wikipedia.org/wiki/Sensor_fusion)

What I wanted to bring up ASAP was when I noticed in linux-next commit bc2b7dab629a51e8beb5fda4222c62a23b729f26 "iio:core: timestamping clock selection support"
that we should probably move this code to e.g. drivers/base or drivers/lib so I can reuse the same ABI in sysfs for GPIO event timestamping.

I think timestamping events uniformly is something that is important to all userspaces.

Quoted 3 lines

Dmitry and I often have to decide on the IIO / input scope boundary (and we've moved it over time) + a number of cross over drivers have turned up recently.

(snip from elsewhere)

Quoted 3 lines

Is it ever sensible to have two drivers for the same part because the use cases are so different? (I hope there is always a better way but maybe not)

I have one on my knee as I want to make a proper IIO driver for MPU-3050 gyroscope which has a (very simplified) driver in drivers/input/misc/mpu3050.c that was merged before the advent of the IIO subsystem.

For the new driver I have these options:

- Keep both drivers and consider them for different usecases. of the same hardware that require different Kconfig (seems very unelegant to me)

- Add some input event calls into the IIO driver so we can phase over the older uses to the new driver.

- Drop the input event handling since gyroscopes should not be handled by input anyway

- Develop a generic IIO-gyroscope-to-input event bridge. (Phew seems complex.)

(elsewhere)

Quoted 1 line

IIO-input has been partly held outside mainline for years by this issue.

Is there such a thing? gitweb?

Quoted 2 lines

1) Are we happy with the somewhat adhoc divisions of subsystems as they stand?

IETF has this motto "rough consensus and running code", and for me that is the working assumption to most things driver-subsystem-division-of-responsibilities-related.

Quoted 5 lines

We have had quite a few cases where a whole driver is submitted to the 'wrong' subsystem. Not the best first interaction with the kernel world for new contributors. (This sort of area is attracts newbies as it's relatively simple and the hardware is fairly cheap and more than once their first experience with review as exactly this!)

What is sometimes lacking is users who are clever enough to take the latest kernel (or preferably even -next) and git grep for a few strings related to the hardware they want to support, and be revealed the fact that there is already a driver for it.

If there isn't, their problem is that of finding the right subsystem to pigeon-hole their hardware into. It typically requires a kernel old grumpy person saying "no, THIS thing goes THERE, THAT thing goes THERE". Unfortunately I think this is pretty hard to codify.

Quoted 1 line

If we aren't happy, what do we do about it?

More helpful kernel generalists...

Quoted 4 lines

GPIO is another interesting case - a lot of hardware is capable of parallel sampling, some at high speeds. It's another area that is probably too specialist for this discussion, but if people want to dive into the details it might be interesting.

I currently have (for v4.8) an ABI which makes i possible to e.g. read 32 simultaneous lines with one userspace/kernelspace swap, if it ends up as a readl() from a register.

The userspace ABI has a limitation to 64 values, so it would be possible to have some driver read 64 GPIO lines and return them in a u8 vector back to userspace in one go.

Quoted 6 lines

For an oscilloscope or signal analyzer usecase with more than 64 channels needing to be sampled simultaneously, we may need to add another "large GPIO read" ABI, and also probably a new driver API interface to achieve it. I've never seen hardware that can read more values in parallel than 32 though, I chose 64 to get some headroom.

Quoted 4 lines

Such a use case would put strong real-time constraints on the system, I wonder how useful a GPIO-based signal analyzer would be. Are you aware of any GPIO hardware that can be programmed to sample at a precise point in time (or perhaps even periodically) ?

I have such hardware. It's custom hardware, non-public at the moment.

But it works like this, for each pin you can configure whether it can be controlled independently, like a normal GPIO, or whether the values are sampled/updated to/from a continuous data stream that is regularly clocked (basically a 1-bit ADC/DAC). This falls right on the edge between GPIO and IIO and I'm really torn where to put it. For now I went with IIO since it predates the chardev GPIO interface and because of all the libiio magic.

Another similar application that uses regular GPIO hardware (IIRC) that falls into the same category is the beaglelogic project
(https://github.com/abhishek-kakkar/BeagleLogic/wiki). And there have been discussions to create a IIO driver for a similar setup in the past.

#19 lore raw
Quoted 7 lines · Lars-Peter Clausen
Quoted 3 lines · Laurent Pinchart

Are you aware of any GPIO
hardware that can be programmed to sample at a precise point in time (or perhaps even periodically) ?

I have such hardware. It's custom hardware, non-public at the moment.

Quoted 7 lines

But it works like this, for each pin you can configure whether it can be controlled independently, like a normal GPIO, or whether the values are sampled/updated to/from a continuous data stream that is regularly clocked (basically a 1-bit ADC/DAC). This falls right on the edge between GPIO and IIO and I'm really torn where to put it. For now I went with IIO since it predates the chardev GPIO interface and because of all the libiio magic.

I guess either GPIO should reuse the IIO triggers, or we create something in IIO that uses GPIO from inside the kernel and add get_multiple() as similar to set_multiple() to the API and thus make a generic GPIO sampler driver in IIO that can reuse the IIO triggers, while using a plain GPIO driver as back-end. If the APIs can be made precise enough.

We can even add get_multiple_with_timestamp() if need be to be able to contain it in gpiolib.

Yours,
Linus Walleij

#20 lore raw
In reply to Laurent Pinchart
Quoted 117 lines · Laurent Pinchart

Hi Linus,

Quoted 107 lines · On Friday 22 Jul 2016 14:04:07 Linus Walleij wrote:
Quoted 4 lines · Jonathan Cameron

This topic would be around the way the various subsystems interact, in the rough area of 'sensors' (I haven't yet had much of an issue with subsystem crossing with output devices but maybe that's just over the next hill!)

Quoted 8 lines

Scope may well be wider but includes:
* input (some of it)
* hwmon
* iio
* comedi(?)
* thermal
* power/battery
* gpio - the blur between gpios and beaglescope / PLC type I/O.

I'm in. The new chardev ABI was designed to fit play/well with IIO, for example I added a 64bit timestamp also to GPIO events so that people could do what I guess they refer to as sensor fusion mixing and blending sensor input events with GPIO events.
(https://en.wikipedia.org/wiki/Sensor_fusion)

What I wanted to bring up ASAP was when I noticed in linux-next commit bc2b7dab629a51e8beb5fda4222c62a23b729f26 "iio:core: timestamping clock selection support"
that we should probably move this code to e.g. drivers/base or drivers/lib so I can reuse the same ABI in sysfs for GPIO event timestamping.

I think timestamping events uniformly is something that is important to all userspaces.

Quoted 3 lines

Dmitry and I often have to decide on the IIO / input scope boundary (and we've moved it over time) + a number of cross over drivers have turned up recently.

(snip from elsewhere)

Quoted 3 lines

Is it ever sensible to have two drivers for the same part because the use cases are so different? (I hope there is always a better way but maybe not)

I have one on my knee as I want to make a proper IIO driver for MPU-3050 gyroscope which has a (very simplified) driver in drivers/input/misc/mpu3050.c that was merged before the advent of the IIO subsystem.

For the new driver I have these options:

- Keep both drivers and consider them for different usecases. of the same hardware that require different Kconfig (seems very unelegant to me)

- Add some input event calls into the IIO driver so we can phase over the older uses to the new driver.

- Drop the input event handling since gyroscopes should not be handled by input anyway

- Develop a generic IIO-gyroscope-to-input event bridge. (Phew seems complex.)

(elsewhere)

Quoted 1 line

IIO-input has been partly held outside mainline for years by this issue.

Is there such a thing? gitweb?

Quoted 2 lines

1) Are we happy with the somewhat adhoc divisions of subsystems as they stand?

IETF has this motto "rough consensus and running code", and for me that is the working assumption to most things driver-subsystem-division-of-responsibilities-related.

Quoted 5 lines

We have had quite a few cases where a whole driver is submitted to the 'wrong' subsystem. Not the best first interaction with the kernel world for new contributors. (This sort of area is attracts newbies as it's relatively simple and the hardware is fairly cheap and more than once their first experience with review as exactly this!)

What is sometimes lacking is users who are clever enough to take the latest kernel (or preferably even -next) and git grep for a few strings related to the hardware they want to support, and be revealed the fact that there is already a driver for it.

If there isn't, their problem is that of finding the right subsystem to pigeon-hole their hardware into. It typically requires a kernel old grumpy person saying "no, THIS thing goes THERE, THAT thing goes THERE". Unfortunately I think this is pretty hard to codify.

Quoted 1 line

If we aren't happy, what do we do about it?

More helpful kernel generalists...

Quoted 4 lines

GPIO is another interesting case - a lot of hardware is capable of parallel sampling, some at high speeds. It's another area that is probably too specialist for this discussion, but if people want to dive into the details it might be interesting.

I currently have (for v4.8) an ABI which makes i possible to e.g. read 32 simultaneous lines with one userspace/kernelspace swap, if it ends up as a readl() from a register.

The userspace ABI has a limitation to 64 values, so it would be possible to have some driver read 64 GPIO lines and return them in a u8 vector back to userspace in one go.

Quoted 6 lines

For an oscilloscope or signal analyzer usecase with more than 64 channels needing to be sampled simultaneously, we may need to add another "large GPIO read" ABI, and also probably a new driver API interface to achieve it. I've never seen hardware that can read more values in parallel than 32 though, I chose 64 to get some headroom.

Quoted 4 lines

Such a use case would put strong real-time constraints on the system, I wonder how useful a GPIO-based signal analyzer would be. Are you aware of any GPIO hardware that can be programmed to sample at a precise point in time (or perhaps even periodically) ?

There are cross over cases from the other side where
people are putting devices that do do sequenced input sampling to use to do gpio tasks. A lot of sensors have spare pins and someone always decides they might as well be gpi(o) pins rather than waste them.

Sometimes they have to be polled separately. Sometimes they have the even more cunning plan of sticking them in 'spare bits' of ADC channels, or in additional registers that can be dropped into a hardware fifo. All in then takes is some designer running low on pins on the SoC to decide they'll be handy for something random.

A simple gpio driven logic analyzer case probably needs some hardware support. There are latched capture units out there that provide this sort of precise timing as well, though they are well out of the realms of what you'd normally call a general purpose input. TIs universal parallel ports for example (I think), or something like beaglelogic where the timing is pushed to a coprocessor.

It's also possible to design a wide variety of algorithms to work with uneven sampling of data, as long as you know 'when' it was sampled.

So more fiddly corner cases :)

Jonathan

Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

Linus Waleij (gpio)

Sure, if travels etc work out and I get invited I'd come running.

#21 lore raw
In reply to Laurent Pinchart
Quoted 9 lines · Laurent Pinchart
Quoted 6 lines · On Friday 22 Jul 2016 14:04:07 Linus Walleij wrote:

For an oscilloscope or signal analyzer usecase with more than 64 channels needing to be sampled simultaneously, we may need to add another "large GPIO read" ABI, and also probably a new driver API interface to achieve it. I've never seen hardware that can read more values in parallel than 32 though, I chose 64 to get some headroom.

Quoted 4 lines

Such a use case would put strong real-time constraints on the system, I wonder how useful a GPIO-based signal analyzer would be. Are you aware of any GPIO hardware that can be programmed to sample at a precise point in time (or perhaps even periodically) ?

Nope. I would expect to close the gap to IIO and reuse their trigger system in that case, instead of creating a new one. IIO uses either hardware trigger interrupts (usually a fixed frequency, some supply hardware timestamps with the interrupts) or alternatively hrtimers, which in turn would I guess be most reliable using RT patches.

Yours,
Linus Walleij

#22 lore raw
In reply to Jonathan Cameron

Hi Jonathan,

Quoted 31 lines · Jonathan Cameron

5) Complex device interaction usecases. At the moment the ones I've come across are mostly contained in IIO. The moment we start sticking in MUXes, AFEs (Analog Front Ends) and straightforward analog sensors in the mix it can get fiddly. Swapping war stories may well be worthwhile on this. This stuff also turns up in ASoC for example so probably lessons to be learned from there.

The analog devices software defined radios are another possible case study.

There of course may well be lessons to be learned from similar interactions elsewhere in the kernel.

There is a lot of history in how we ended up where we are (it all made absolute sense at the time). Sitting back and taking the time to discuss the future would be great. Whilst this might be solvable by email we've made no definitive progress for years
(and what has been made has been on a case by case basis deep in driver reviews.)

I threw comedi in the list above but, at the moment, I think the more likely direction there is a single userspace library abstracting the underlying subsystem (Analog Devices are working in that direction - perhaps Lars can offer more on that?).

GPIO is another interesting case - a lot of hardware is capable of parallel sampling, some at high speeds. It's another area that is probably too specialist for this discussion, but if people want to dive into the details it might be interesting.

Quoted 7 lines

I think we have only a small amount of fuzz around the v4l boundary, but wanted to leave the door open if anyone wants to discuss that one further as it's come up a few times over recent years.

The SoC world is a major case of one device, many uses. Some SoCs
are turning up with multiple ADC units, sometimes with different designs, sometimes simply so that the same hardware can be used for different things.

I would be interested in this discussion. Complex device seem to be getting more complex over here as well!

Thanks

Signature · 1 line
~Vinod
#23 lore raw
In reply to Jonathan Cameron

Em Wed, 20 Jul 2016 22:18:11 +0100
Jonathan Cameron <[email protected]> escreveu:

Quoted 5 lines

Hi All,

This topic would be around the way the various subsystems interact, in the rough area of 'sensors' (I haven't yet had much of an issue with subsystem crossing with output devices but maybe that's just over the next hill!)

Yeah, there is a gray area here as devices become more complex. So, I'm interested on such topic.

Quoted 11 lines

Scope may well be wider but includes:
* input (some of it)
* hwmon
* iio
* comedi(?)
* thermal
* power/battery
* gpio - the blur between gpios and beaglescope / PLC type I/O.
* v4l - when does a device jump from being a multi pixel thermopile to a thermal camera? Smart fingerprint scanners etc. Gesture recognition sensors (effectively 9ish pixel cameras)

For devices that provide images somehow, I'd say that the best would be to implement them as via the V4L2 API. We'll likely need to discuss it case by case, specially for early cross-subsystem drivers, though, as it is not trivial to identify the subsystem boundaries, and sometimes multiple APIs from different subsystems is needed. Also, it is not clear where such drivers would fit at the Kernel tree.

One interesting case is an input device driver with multi-finger support (drivers/input/touchscreen/sur40.c). I suspect we'll have more cases like that over time.

Quoted 1 line

* Lots of random things we haven't seen yet.

Thanks,
Mauro

#24 lore raw
Quoted 22 lines · On Thursday 28 Jul 2016 08:58:46 Mauro Carvalho Chehab wrote:

Em Wed, 20 Jul 2016 22:18:11 +0100 Jonathan Cameron escreveu:

Quoted 5 lines

Hi All,

This topic would be around the way the various subsystems interact, in the rough area of 'sensors' (I haven't yet had much of an issue with subsystem crossing with output devices but maybe that's just over the next hill!)

Yeah, there is a gray area here as devices become more complex. So, I'm interested on such topic.

Quoted 11 lines

Scope may well be wider but includes:
* input (some of it)
* hwmon
* iio
* comedi(?)
* thermal
* power/battery
* gpio - the blur between gpios and beaglescope / PLC type I/O.
* v4l - when does a device jump from being a multi pixel thermopile to a thermal camera? Smart fingerprint scanners etc. Gesture recognition sensors (effectively 9ish pixel cameras)

Quoted 2 lines

For devices that provide images somehow, I'd say that the best would be to implement them as via the V4L2 API.

This requires defining what an image is. Furthermore, we very low image resolution devices, we will need to deal with high frame rates (1k-10k is a range we need to consider). The V4L2 API will likely show performance issues.

Quoted 10 lines

We'll likely need to discuss it case by case, specially for early cross- subsystem drivers, though, as it is not trivial to identify the subsystem boundaries, and sometimes multiple APIs from different subsystems is needed. Also, it is not clear where such drivers would fit at the Kernel tree.

One interesting case is an input device driver with multi-finger support (drivers/input/touchscreen/sur40.c). I suspect we'll have more cases like that over time.

Quoted 1 line

* Lots of random things we haven't seen yet.

Signature · 3 lines
Regards,

Laurent Pinchart
#25 lore raw
Quoted 23 lines · Laurent Pinchart
Quoted 22 lines · On Thursday 28 Jul 2016 08:58:46 Mauro Carvalho Chehab wrote:

Em Wed, 20 Jul 2016 22:18:11 +0100 Jonathan Cameron escreveu:

Quoted 5 lines

Hi All,

This topic would be around the way the various subsystems interact, in the rough area of 'sensors' (I haven't yet had much of an issue with subsystem crossing with output devices but maybe that's just over the next hill!)

Yeah, there is a gray area here as devices become more complex. So, I'm interested on such topic.

Quoted 11 lines

Scope may well be wider but includes:
* input (some of it)
* hwmon
* iio
* comedi(?)
* thermal
* power/battery
* gpio - the blur between gpios and beaglescope / PLC type I/O.
* v4l - when does a device jump from being a multi pixel thermopile to a thermal camera? Smart fingerprint scanners etc. Gesture recognition sensors (effectively 9ish pixel cameras)

Quoted 6 lines
Quoted 2 lines

For devices that provide images somehow, I'd say that the best would be to implement them as via the V4L2 API.

This requires defining what an image is. Furthermore, we very low image resolution devices, we will need to deal with high frame rates (1k-10k is a range we need to consider). The V4L2 API will likely show performance issues.

An interesting point I'd never thought about before. Funnily enough the devices around this boundary that I've encountered have all been relatively slow. Doubt we'll be lucky enough that that will last!

Jonathan

Quoted 12 lines
Quoted 11 lines

We'll likely need to discuss it case by case, specially for early cross- subsystem drivers, though, as it is not trivial to identify the subsystem boundaries, and sometimes multiple APIs from different subsystems is needed. Also, it is not clear where such drivers would fit at the Kernel tree.

One interesting case is an input device driver with multi-finger support (drivers/input/touchscreen/sur40.c). I suspect we'll have more cases like that over time.

Quoted 1 line

* Lots of random things we haven't seen yet.

#26 lore raw
Quoted 28 lines · On Thursday 28 Jul 2016 23:09:59 Jonathan Cameron wrote:
Quoted 27 lines · Laurent Pinchart
Quoted 25 lines · On Thursday 28 Jul 2016 08:58:46 Mauro Carvalho Chehab wrote:

Em Wed, 20 Jul 2016 22:18:11 +0100 Jonathan Cameron escreveu:

Quoted 6 lines

Hi All,

This topic would be around the way the various subsystems interact, in the rough area of 'sensors' (I haven't yet had much of an issue with subsystem crossing with output devices but maybe that's just over the next hill!)

Yeah, there is a gray area here as devices become more complex. So, I'm interested on such topic.

Quoted 11 lines

Scope may well be wider but includes:
* input (some of it)
* hwmon
* iio
* comedi(?)
* thermal
* power/battery
* gpio - the blur between gpios and beaglescope / PLC type I/O.
* v4l - when does a device jump from being a multi pixel thermopile to a thermal camera? Smart fingerprint scanners etc. Gesture recognition sensors (effectively 9ish pixel cameras)

For devices that provide images somehow, I'd say that the best would be to implement them as via the V4L2 API.

Quoted 8 lines
Quoted 4 lines

This requires defining what an image is. Furthermore, we very low image resolution devices, we will need to deal with high frame rates (1k-10k is a range we need to consider). The V4L2 API will likely show performance issues.

An interesting point I'd never thought about before. Funnily enough the devices around this boundary that I've encountered have all been relatively slow. Doubt we'll be lucky enough that that will last!

For the sake of completeness, I'd like to add that always-on image sensors
(for instance https://globenewswire.com/news-release/2016/01/19/802804/0/en/Himax-Launches-Ultra-Low-Power-CMOS-Image-Sensor-for-Always-On-Computer-Vision-Applications.html) will also bring interesting challenges. The frame rate won't be particularly high, but the low
resolution will allow transmitting image data over I2C (or rather I3C, a
faster I2C-compatible serial bus, see http://mipi.org/specifications/i3c℠-sensor-specification) in a continuous fashion.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

We'll likely need to discuss it case by case, specially for early cross- subsystem drivers, though, as it is not trivial to identify the subsystem boundaries, and sometimes multiple APIs from different subsystems is needed. Also, it is not clear where such drivers would fit at the Kernel tree.

One interesting case is an input device driver with multi-finger support (drivers/input/touchscreen/sur40.c). I suspect we'll have more cases like that over time.

Quoted 1 line

* Lots of random things we haven't seen yet.

Signature · 3 lines
Regards,

Laurent Pinchart
#27 lore raw
In reply to Laurent Pinchart
Quoted 23 lines · Laurent Pinchart
Quoted 22 lines · On Thursday 28 Jul 2016 08:58:46 Mauro Carvalho Chehab wrote:

Em Wed, 20 Jul 2016 22:18:11 +0100 Jonathan Cameron escreveu:

Quoted 5 lines

Hi All,

This topic would be around the way the various subsystems interact, in the rough area of 'sensors' (I haven't yet had much of an issue with subsystem crossing with output devices but maybe that's just over the next hill!)

Yeah, there is a gray area here as devices become more complex. So, I'm interested on such topic.

Quoted 11 lines

Scope may well be wider but includes:
* input (some of it)
* hwmon
* iio
* comedi(?)
* thermal
* power/battery
* gpio - the blur between gpios and beaglescope / PLC type I/O.
* v4l - when does a device jump from being a multi pixel thermopile to a thermal camera? Smart fingerprint scanners etc. Gesture recognition sensors (effectively 9ish pixel cameras)

Quoted 6 lines
Quoted 2 lines

For devices that provide images somehow, I'd say that the best would be to implement them as via the V4L2 API.

This requires defining what an image is. Furthermore, we very low image resolution devices, we will need to deal with high frame rates (1k-10k is a range we need to consider). The V4L2 API will likely show performance issues.

The main problems there are the limit to the number of buffers (32) in v4l2 and that the number of qbuf/dqbuf ioctls is large for these high framerates. You would probably want to introduce mqbuf/mdqbuf ioctls to queue/dequeue multiple buffers. The core internals of v4l2 (vb2) are all pretty low overhead.

Regards,

Hans

Quoted 12 lines
Quoted 11 lines

We'll likely need to discuss it case by case, specially for early cross- subsystem drivers, though, as it is not trivial to identify the subsystem boundaries, and sometimes multiple APIs from different subsystems is needed. Also, it is not clear where such drivers would fit at the Kernel tree.

One interesting case is an input device driver with multi-finger support (drivers/input/touchscreen/sur40.c). I suspect we'll have more cases like that over time.

Quoted 1 line

* Lots of random things we haven't seen yet.

#28 lore raw

Hi Hans,

Quoted 33 lines · On Friday 29 Jul 2016 09:28:06 Hans Verkuil wrote:
Quoted 24 lines · Laurent Pinchart
Quoted 23 lines · On Thursday 28 Jul 2016 08:58:46 Mauro Carvalho Chehab wrote:

Em Wed, 20 Jul 2016 22:18:11 +0100 Jonathan Cameron escreveu:

Quoted 6 lines

Hi All,

This topic would be around the way the various subsystems interact, in the rough area of 'sensors' (I haven't yet had much of an issue with subsystem crossing with output devices but maybe that's just over the next hill!)

Yeah, there is a gray area here as devices become more complex. So, I'm interested on such topic.

Quoted 11 lines

Scope may well be wider but includes:
* input (some of it)
* hwmon
* iio
* comedi(?)
* thermal
* power/battery
* gpio - the blur between gpios and beaglescope / PLC type I/O.
* v4l - when does a device jump from being a multi pixel thermopile to a thermal camera? Smart fingerprint scanners etc. Gesture recognition sensors (effectively 9ish pixel cameras)

Quoted 7 lines
Quoted 2 lines

For devices that provide images somehow, I'd say that the best would be to implement them as via the V4L2 API.

This requires defining what an image is. Furthermore, we very low image resolution devices, we will need to deal with high frame rates (1k-10k is a range we need to consider). The V4L2 API will likely show performance issues.

Quoted 5 lines

The main problems there are the limit to the number of buffers (32) in v4l2 and that the number of qbuf/dqbuf ioctls is large for these high framerates. You would probably want to introduce mqbuf/mdqbuf ioctls to queue/dequeue multiple buffers. The core internals of v4l2 (vb2) are all pretty low overhead.

Or, just brainstorming, using some kind of ring-buffer based API that doesn't require context switches between userspace and kernelspace. That could also be somehow useful for larger images, as low-latency video streaming could benefit from the ability to start encoding the image before it's fully captured (or the other way around, we could start consuming it before it's fully decoded).

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

We'll likely need to discuss it case by case, specially for early cross- subsystem drivers, though, as it is not trivial to identify the subsystem boundaries, and sometimes multiple APIs from different subsystems is needed. Also, it is not clear where such drivers would fit at the Kernel tree.

One interesting case is an input device driver with multi-finger support (drivers/input/touchscreen/sur40.c). I suspect we'll have more cases like that over time.

Quoted 1 line

* Lots of random things we haven't seen yet.

Signature · 3 lines
Regards,

Laurent Pinchart
#29 lore raw
In reply to Mauro Carvalho Chehab
Quoted 24 lines · Mauro Carvalho Chehab

Em Wed, 20 Jul 2016 22:18:11 +0100
Jonathan Cameron <[email protected]> escreveu:

Quoted 5 lines

Hi All,

This topic would be around the way the various subsystems interact, in the rough area of 'sensors' (I haven't yet had much of an issue with subsystem crossing with output devices but maybe that's just over the next hill!)

Yeah, there is a gray area here as devices become more complex. So, I'm interested on such topic.

Quoted 11 lines

Scope may well be wider but includes:
* input (some of it)
* hwmon
* iio
* comedi(?)
* thermal
* power/battery
* gpio - the blur between gpios and beaglescope / PLC type I/O.
* v4l - when does a device jump from being a multi pixel thermopile to a thermal camera? Smart fingerprint scanners etc. Gesture recognition sensors (effectively 9ish pixel cameras)

Quoted 6 lines

For devices that provide images somehow, I'd say that the best would be to implement them as via the V4L2 API. We'll likely need to discuss it case by case, specially for early cross-subsystem drivers, though, as it is not trivial to identify the subsystem boundaries, and sometimes multiple APIs from different subsystems is needed. Also, it is not clear where such drivers would fit at the Kernel tree.

Absolutely. We can come up with some rule's of thumb to at least cut down on the number of drivers getting written / pitched for totally the wrong subsystem, but there are always going to be corner cases.

Quoted 4 lines

One interesting case is an input device driver with multi-finger support (drivers/input/touchscreen/sur40.c). I suspect we'll have more cases like that over time.

Becomes a case of whether we almost 'limit' devices by putting them in a subsystem dictated by their usecase, or find a way to bridge between subsystems so the common use case and interfaces are supported but the more 'interesting' stuff is available underneath - possibly for other uses.

Quoted 2 lines
Quoted 2 lines

* Lots of random things we haven't seen yet.

I've already seen a few in this thread!

Jonathan

Quoted 4 lines

Thanks,
Mauro

#30 lore raw
In reply to Jonathan Cameron

Hi Jonathan,

Quoted 5 lines · On Wednesday 20 Jul 2016 22:18:11 Jonathan Cameron wrote:

Hi All,

This topic would be around the way the various subsystems interact, in the rough area of 'sensors' (I haven't yet had much of an issue with subsystem crossing with output devices but maybe that's just over the next hill!)

It's already here :-) We have an API to control camera flash (as in light, not memory) in V4L2 and another one in the LED subsystem.

Quoted 56 lines

Scope may well be wider but includes:
* input (some of it)
* hwmon
* iio
* comedi(?)
* thermal
* power/battery
* gpio - the blur between gpios and beaglescope / PLC type I/O.
* v4l - when does a device jump from being a multi pixel thermopile to a thermal camera? Smart fingerprint scanners etc. Gesture recognition sensors (effectively 9ish pixel cameras)
* Lots of random things we haven't seen yet.

All these make use of devices that are at their hearts ADCs. Whilst it would nice if the underlying devices always fell into one clear category that is not always the case. Hardware descriptions (e.g. device tree) are an important area of contention.

The examples that follow are mostly based around IIO interactions with hwmon / input as that's my area of expertise. However, I know that similar issues occur at other boundaries.

Guenter Roeck has been working on hwmon / thermal interfacing recently. We also regularly push drivers one way or the other around that boundary.

Dmitry and I often have to decide on the IIO / input scope boundary (and we've moved it over time) + a number of cross over drivers have turned up recently.

The other subsystems I encounter have been 'quieter' in their interactions with me but many of the issues below apply to them as well.

A rough list of related topics that might be worth face to face discussion includes:

1) Are we happy with the somewhat adhoc divisions of subsystems as they stand?

I am personally happy enough with this, but it's worth checking everyone else is. This probably causes more pain for new submitters than it does for old hands. Perhaps we can come up with a checklist style doc to save people having to ask.

We have had quite a few cases where a whole driver is submitted to the 'wrong' subsystem. Not the best first interaction with the kernel world for new contributors. (This sort of area is attracts newbies as it's relatively simple and the hardware is fairly cheap and more than once their first experience with review as exactly this!)

If we aren't happy, what do we do about it?

I'm hoping we are happy or, at least, resigned to the current state of affairs, but think the question should be asked from time to time as the answer may well change + the discussion will highlight pinch points we can work on.

Quoted 8 lines

2) Bridging drivers - there will always be cross over cases. E.g. Generic ADCs used for battery voltage monitoring. We have a number of bridges in mainline already and others out of tree. How important are these and what features would make them more useful?

Is it ever sensible to have two drivers for the same part because the use cases are so different? (I hope there is always a better way but maybe not)

I'd love to say no, but we have already hit that problem. There are two drivers for the ADV7511 HDMI encoder, one in the media subsystem for V4L2 output devices (drivers/media/i2c/adv7511.c), and one in the DRM/KMS subsystem for display devices (drivers/gpu/drm/i2c/adv7511.c, being moved to drivers/gpu/drm/bridge/). Try compiling them both in the kernel and see how they race to probe the same device (enjoy the show, popcorns not included).

Quoted 56 lines

This covers both generic bridges (e.g. iio-hwmon) and device specific bridges (a good example of a touch screen driver turned up in my inbox yesterday).

3) Bindings (device tree and similar). When it is appropriate to use a device tree to describe the overlap (bridging of channels)? Sometimes there is obvious real hardware involved (a thermistor on a generic ADC input for example), but there are many grey areas.

Finally bashing out an answer to the issue device tree maintainers have with the iio-hwmon bindings would be great.
On a less biased note, that example is pretty fiddly to define but would act as a good basis to work from more generally.

We know we got it wrong (back in the dark ages), but it isn't obvious how to get it right!

IIO-input has been partly held outside mainline for years by this issue.

Also some practical design decisions to make around how to implement these mappings to work best with deferred probing etc.

4) Should we drop all the bindings for bridging between such subsystems and do it all from userspace?

I think we may end up with a hybrid of the two, but need to be able to make it work in 'standard' cases without userspace being involved. That hybrid solution may well be devicetree overlay based... Unclear so far.
+ plenty of crazy things that 'might' make sense where there isn't even the pretence of representing real hardware.

5) Complex device interaction usecases. At the moment the ones I've come across are mostly contained in IIO. The moment we start sticking in MUXes, AFEs (Analog Front Ends) and straightforward analog sensors in the mix it can get fiddly. Swapping war stories may well be worthwhile on this. This stuff also turns up in ASoC for example so probably lessons to be learned from there.

The analog devices software defined radios are another possible case study.

There of course may well be lessons to be learned from similar interactions elsewhere in the kernel.

There is a lot of history in how we ended up where we are (it all made absolute sense at the time). Sitting back and taking the time to discuss the future would be great. Whilst this might be solvable by email we've made no definitive progress for years
(and what has been made has been on a case by case basis deep in driver reviews.)

I threw comedi in the list above but, at the moment, I think the more likely direction there is a single userspace library abstracting the underlying subsystem (Analog Devices are working in that direction - perhaps Lars can offer more on that?).

Quoted 8 lines

GPIO is another interesting case - a lot of hardware is capable of parallel sampling, some at high speeds. It's another area that is probably too specialist for this discussion, but if people want to dive into the details it might be interesting.

I think we have only a small amount of fuzz around the v4l boundary, but wanted to leave the door open if anyone wants to discuss that one further as it's come up a few times over recent years.

Don't forget to take system integration into account. If I given you a high- speed ADC you will not think about V4L2 as your subsystem of choice. If the system designer has connected that ADC to a CPLD that generates fake horizontal and vertical sync signals, and connected the output to the camera interface of the SoC, you will be left with no choice but use the V4L2 API. That's largely a userspace issue in this case, but it implies that V4L2 need to define an "image format" for the ADC data.

Quoted 24 lines

The SoC world is a major case of one device, many uses. Some SoCs are turning up with multiple ADC units, sometimes with different designs, sometimes simply so that the same hardware can be used for different things.

A few suggestions for people:

Guenter Roeck (hmwon)
Dmitry Torokhov (input)
Me (Jonathan Cameron - IIO)

Someone thermal related (not sure who would be most interested) Someone power supply related? (Guenter, any suggestions?)

Someone device tree related (Mark Rutland? Rob Herring?)

Linus Waleij (gpio)

Mark Brown - as some of concepts of IIO bridging and how far it can be taken (everything via a bridge driver) came from discussions with him on SoC ADC handling.

Lar-Peter Clausen for generally doing insane things with ADCs + having one foot in the userspace side of things.

Signature · 3 lines
Regards,

Laurent Pinchart
#31 lore raw

On 07/28/2016 06:39 PM, Laurent Pinchart wrote:
[...]

Quoted 11 lines
Quoted 3 lines

I think we have only a small amount of fuzz around the v4l boundary, but wanted to leave the door open if anyone wants to discuss that one further as it's come up a few times over recent years.

Don't forget to take system integration into account. If I given you a high- speed ADC you will not think about V4L2 as your subsystem of choice. If the system designer has connected that ADC to a CPLD that generates fake horizontal and vertical sync signals, and connected the output to the camera interface of the SoC, you will be left with no choice but use the V4L2 API. That's largely a userspace issue in this case, but it implies that V4L2 need to define an "image format" for the ADC data.

I think this hits the core of this discussion. Todays hardware is getting more and more generic. It is a lot more economical to produce a single general purpose high-volume device than a handful of low or medium volume specialized devices. Even if the raw production cost of the general purpose part will be higher (since it contains more logic) the overall per unit price will be lower since the per part contribution of the one-time design cost is lower in a high volume run.

So new hardware often tends to be general purpose and can be used in many different applications.

But our kernel frameworks are designed around application specific tasks.

* ALSA is for audio data capture/playback
* V4L2 is for video data capture/playback
* DRM is for video display
* IIO is for sensor data capture/playback

When you capture data over a particular interface there is a specific meaning associated to the data rather than the data just being data, which is how the hardware might see it.

On the kernel side we have started to address this by having generic frameworks like DMAengine. I've used the same DMA core with the same DMAengine driver exposed to userspace for all four different types listed above depending on the application.

This works as long as you know that your hardware is generic and you design the driver to be generic. But it breaks if your hardware has a primary function that is application specific.

E.g. a CSI-2 receiver will most likely receive video data, so we write a V4L2 driver for it. A I2S receiver will most likely receive audio data, so we write a ALSA driver for it. But now somebody might decide to hook up a gyroscope to any of these interfaces because that might be the best way to feed data into the particular SoC used in that system. And than things start to fall apart.

And this is not just hypothetical I've seen questions how to make this work repeatedly. And I also expect that in a time constraint environment people will go ahead with a custom solution where they capture audio data through V4L2 and just ignore all data type hints V4L2 provides and re-interprets the data since their specialized application knows what the data layout looks like.

A similar issue is that there are quite a few pieces of hardware that are multi-use. E.g. general purpose serial data cores that support SPI, I2S, and similar. At the moment we have to write two different drivers for them using compatible strings to decide which function they should have.

So going forward we might have to address this by creating a more generic interface that allows us to exchange data between a peripheral and a application without assigning any kind of meaning to the data itself. And then have that meaning provided through side channels. E.g. a V4L2 device could say this over there is my data capture device and the data layout is the following. Similar for the other frameworks that allow capture/playback.

With vb2 (former V4L2 buffer handling code) now being independent from the V4L2 framework this might be a prime candidate as a starting point. I've been meaning to re-write the IIO DMA buffer code on top of vb2 to reduce the amount of custom code.

This of course would be a very grand task and maybe we'll loose ourselves in endless discussions about the details and all the corner cases that need to be considered. But if we want to find a solution that keeps up with the current trend that the hardware landscape seems to be going in we might have no other choice. Otherwise I'd say it is inevitable that we see more and more hardware which has multiple drivers, each driver handling a different type of application.

Such a grand unified media framework would also help for applications where multiple streams of different data types need to be synchronized e.g. audio and video.

- Lars

#32 lore raw
Quoted 6 lines · Lars-Peter Clausen

But our kernel frameworks are designed around application specific tasks.

* ALSA is for audio data capture/playback
* V4L2 is for video data capture/playback
* DRM is for video display
* IIO is for sensor data capture/playback

Plus as we were discussing elsewhere hwmon, power_supply, input and probably others that aren't springing to mind right now that also represent sensor values out to userspace.

Quoted 6 lines

And this is not just hypothetical I've seen questions how to make this work repeatedly. And I also expect that in a time constraint environment people will go ahead with a custom solution where they capture audio data through V4L2 and just ignore all data type hints V4L2 provides and re-interprets the data since their specialized application knows what the data layout looks like.

I've also seen cases (indeed I've got one myself right now) where the opposite case applies and it's a lot easier to put a chip into a hardware class it wasn't really designed for because the userspace interfaces we offer are much more appropriate for that application. In that case it's a chip that really wants to be hwmon but libiio is *really* nice.

Quoted 7 lines

So going forward we might have to address this by creating a more generic interface that allows us to exchange data between a peripheral and a application without assigning any kind of meaning to the data itself. And then have that meaning provided through side channels. E.g. a V4L2 device could say this over there is my data capture device and the data layout is the following. Similar for the other frameworks that allow capture/playback.

Those side channels are the most tricky bit I fear.

Quoted 7 lines

This of course would be a very grand task and maybe we'll loose ourselves in endless discussions about the details and all the corner cases that need to be considered. But if we want to find a solution that keeps up with the current trend that the hardware landscape seems to be going in we might have no other choice. Otherwise I'd say it is inevitable that we see more and more hardware which has multiple drivers, each driver handling a different type of application.

It's a worthwhile goal to bring things together. I wonder how hard it'd be to get ALSA to interact with the v4l buffer style...

#33 lore raw
In reply to Lars-Peter Clausen
Quoted 9 lines · Lars-Peter Clausen

So new hardware often tends to be general purpose and can be used in many different applications.

But our kernel frameworks are designed around application specific tasks.

* ALSA is for audio data capture/playback
* V4L2 is for video data capture/playback
* DRM is for video display
* IIO is for sensor data capture/playback

Spot on. Even if you look at kernel source code we have directory based on end usage..

Also these subsystems expose a user interface for sending/receiving data. The interface is application specific. One can argue that designing a generic data interface is kernel which can take care of high volume, bursty or continuous data, then potentially we can unify these

Quoted 8 lines

When you capture data over a particular interface there is a specific meaning associated to the data rather than the data just being data, which is how the hardware might see it.

On the kernel side we have started to address this by having generic frameworks like DMAengine. I've used the same DMA core with the same DMAengine driver exposed to userspace for all four different types listed above depending on the application.

Yes DMAengine framework is generic for application usage. We don't define the purpose of DMA but we care about nature, memcpy, slave, interleaved...

You can use these for audio, video, RAID or anything else..

Quoted 7 lines

This of course would be a very grand task and maybe we'll loose ourselves in endless discussions about the details and all the corner cases that need to be considered. But if we want to find a solution that keeps up with the current trend that the hardware landscape seems to be going in we might have no other choice. Otherwise I'd say it is inevitable that we see more and more hardware which has multiple drivers, each driver handling a different type of application.

Given that SoCs are becoming complex and HW folks are merging stuff, I wont be surprised if we get a converged hardware which does multiple jobs, so we would need these to be unified.

Signature · 1 line
~Vinod
#34 lore raw
In reply to Lars-Peter Clausen

Hi Lars,

Quoted 79 lines · On Thursday 28 Jul 2016 20:53:26 Lars-Peter Clausen wrote:

On 07/28/2016 06:39 PM, Laurent Pinchart wrote:
[...]

Quoted 4 lines
Quoted 3 lines

I think we have only a small amount of fuzz around the v4l boundary, but wanted to leave the door open if anyone wants to discuss that one further as it's come up a few times over recent years.

Quoted 8 lines

Don't forget to take system integration into account. If I given you a high- speed ADC you will not think about V4L2 as your subsystem of choice. If the system designer has connected that ADC to a CPLD that generates fake horizontal and vertical sync signals, and connected the output to the camera interface of the SoC, you will be left with no choice but use the V4L2 API. That's largely a userspace issue in this case, but it implies that V4L2 need to define an "image format" for the ADC data.

I think this hits the core of this discussion. Todays hardware is getting more and more generic. It is a lot more economical to produce a single general purpose high-volume device than a handful of low or medium volume specialized devices. Even if the raw production cost of the general purpose part will be higher (since it contains more logic) the overall per unit price will be lower since the per part contribution of the one-time design cost is lower in a high volume run.

So new hardware often tends to be general purpose and can be used in many different applications.

But our kernel frameworks are designed around application specific tasks.

* ALSA is for audio data capture/playback
* V4L2 is for video data capture/playback
* DRM is for video display
* IIO is for sensor data capture/playback

When you capture data over a particular interface there is a specific meaning associated to the data rather than the data just being data, which is how the hardware might see it.

On the kernel side we have started to address this by having generic frameworks like DMAengine. I've used the same DMA core with the same DMAengine driver exposed to userspace for all four different types listed above depending on the application.

This works as long as you know that your hardware is generic and you design the driver to be generic. But it breaks if your hardware has a primary function that is application specific.

E.g. a CSI-2 receiver will most likely receive video data, so we write a V4L2 driver for it. A I2S receiver will most likely receive audio data, so we write a ALSA driver for it. But now somebody might decide to hook up a gyroscope to any of these interfaces because that might be the best way to feed data into the particular SoC used in that system. And than things start to fall apart.

And this is not just hypothetical I've seen questions how to make this work repeatedly. And I also expect that in a time constraint environment people will go ahead with a custom solution where they capture audio data through V4L2 and just ignore all data type hints V4L2 provides and re-interprets the data since their specialized application knows what the data layout looks like.

A similar issue is that there are quite a few pieces of hardware that are multi-use. E.g. general purpose serial data cores that support SPI, I2S, and similar. At the moment we have to write two different drivers for them using compatible strings to decide which function they should have.

So going forward we might have to address this by creating a more generic interface that allows us to exchange data between a peripheral and a application without assigning any kind of meaning to the data itself. And then have that meaning provided through side channels. E.g. a V4L2 device could say this over there is my data capture device and the data layout is the following. Similar for the other frameworks that allow capture/playback.

With vb2 (former V4L2 buffer handling code) now being independent from the V4L2 framework this might be a prime candidate as a starting point. I've been meaning to re-write the IIO DMA buffer code on top of vb2 to reduce the amount of custom code.

Quoted 7 lines

This of course would be a very grand task and maybe we'll loose ourselves in endless discussions about the details and all the corner cases that need to be considered. But if we want to find a solution that keeps up with the current trend that the hardware landscape seems to be going in we might have no other choice. Otherwise I'd say it is inevitable that we see more and more hardware which has multiple drivers, each driver handling a different type of application.

You have a very good point here. It might be time decoupling the control and data parts of our userspace APIs. The buffer queued-based method of data passing is very similar between V4L2 and IIO as far as I understand, there might not be any good reason to have two APIs there apart from historical reasons. When it comes to controlling the device, though, video and I/O are pretty different.

If we were designing this from scratch today tt could thus make sense to standardize cross-subsystem methods for passing large quantities of data between kernelspace and userspace (buffer queues and ring buffers come to mind), with a set of common API elements (most likely IOCTLs). As we have to deal with our history I'm not sure what latitude we still have to fix the problem.

Quoted 3 lines

Such a grand unified media framework would also help for applications where multiple streams of different data types need to be synchronized e.g. audio and video.

Signature · 3 lines
Regards,

Laurent Pinchart
#35 lore raw
In reply to Laurent Pinchart
Quoted 11 lines · Laurent Pinchart

Hi Jonathan,

Quoted 5 lines · On Wednesday 20 Jul 2016 22:18:11 Jonathan Cameron wrote:

Hi All,

This topic would be around the way the various subsystems interact, in the rough area of 'sensors' (I haven't yet had much of an issue with subsystem crossing with output devices but maybe that's just over the next hill!)

It's already here :-) We have an API to control camera flash (as in light, not memory) in V4L2 and another one in the LED subsystem.

I wonder how many devices are doing it with a gpio or some of the more configurable PWM units..

Quoted 66 lines
Quoted 57 lines

Scope may well be wider but includes:
* input (some of it)
* hwmon
* iio
* comedi(?)
* thermal
* power/battery
* gpio - the blur between gpios and beaglescope / PLC type I/O.
* v4l - when does a device jump from being a multi pixel thermopile to a thermal camera? Smart fingerprint scanners etc. Gesture recognition sensors (effectively 9ish pixel cameras)
* Lots of random things we haven't seen yet.

All these make use of devices that are at their hearts ADCs. Whilst it would nice if the underlying devices always fell into one clear category that is not always the case. Hardware descriptions (e.g. device tree) are an important area of contention.

The examples that follow are mostly based around IIO interactions with hwmon / input as that's my area of expertise. However, I know that similar issues occur at other boundaries.

Guenter Roeck has been working on hwmon / thermal interfacing recently. We also regularly push drivers one way or the other around that boundary.

Dmitry and I often have to decide on the IIO / input scope boundary (and we've moved it over time) + a number of cross over drivers have turned up recently.

The other subsystems I encounter have been 'quieter' in their interactions with me but many of the issues below apply to them as well.

A rough list of related topics that might be worth face to face discussion includes:

1) Are we happy with the somewhat adhoc divisions of subsystems as they stand?

I am personally happy enough with this, but it's worth checking everyone else is. This probably causes more pain for new submitters than it does for old hands. Perhaps we can come up with a checklist style doc to save people having to ask.

We have had quite a few cases where a whole driver is submitted to the 'wrong' subsystem. Not the best first interaction with the kernel world for new contributors. (This sort of area is attracts newbies as it's relatively simple and the hardware is fairly cheap and more than once their first experience with review as exactly this!)

If we aren't happy, what do we do about it?

I'm hoping we are happy or, at least, resigned to the current state of affairs, but think the question should be asked from time to time as the answer may well change + the discussion will highlight pinch points we can work on.

Quoted 8 lines

2) Bridging drivers - there will always be cross over cases. E.g. Generic ADCs used for battery voltage monitoring. We have a number of bridges in mainline already and others out of tree. How important are these and what features would make them more useful?

Is it ever sensible to have two drivers for the same part because the use cases are so different? (I hope there is always a better way but maybe not)

Quoted 6 lines

I'd love to say no, but we have already hit that problem. There are two drivers for the ADV7511 HDMI encoder, one in the media subsystem for V4L2 output devices (drivers/media/i2c/adv7511.c), and one in the DRM/KMS subsystem for display devices (drivers/gpu/drm/i2c/adv7511.c, being moved to drivers/gpu/drm/bridge/). Try compiling them both in the kernel and see how they race to probe the same device (enjoy the show, popcorns not included).

Nasty.

Quoted 66 lines
Quoted 57 lines

This covers both generic bridges (e.g. iio-hwmon) and device specific bridges (a good example of a touch screen driver turned up in my inbox yesterday).

3) Bindings (device tree and similar). When it is appropriate to use a device tree to describe the overlap (bridging of channels)? Sometimes there is obvious real hardware involved (a thermistor on a generic ADC input for example), but there are many grey areas.

Finally bashing out an answer to the issue device tree maintainers have with the iio-hwmon bindings would be great.
On a less biased note, that example is pretty fiddly to define but would act as a good basis to work from more generally.

We know we got it wrong (back in the dark ages), but it isn't obvious how to get it right!

IIO-input has been partly held outside mainline for years by this issue.

Also some practical design decisions to make around how to implement these mappings to work best with deferred probing etc.

4) Should we drop all the bindings for bridging between such subsystems and do it all from userspace?

I think we may end up with a hybrid of the two, but need to be able to make it work in 'standard' cases without userspace being involved. That hybrid solution may well be devicetree overlay based... Unclear so far.
+ plenty of crazy things that 'might' make sense where there isn't even the pretence of representing real hardware.

5) Complex device interaction usecases. At the moment the ones I've come across are mostly contained in IIO. The moment we start sticking in MUXes, AFEs (Analog Front Ends) and straightforward analog sensors in the mix it can get fiddly. Swapping war stories may well be worthwhile on this. This stuff also turns up in ASoC for example so probably lessons to be learned from there.

The analog devices software defined radios are another possible case study.

There of course may well be lessons to be learned from similar interactions elsewhere in the kernel.

There is a lot of history in how we ended up where we are (it all made absolute sense at the time). Sitting back and taking the time to discuss the future would be great. Whilst this might be solvable by email we've made no definitive progress for years
(and what has been made has been on a case by case basis deep in driver reviews.)

I threw comedi in the list above but, at the moment, I think the more likely direction there is a single userspace library abstracting the underlying subsystem (Analog Devices are working in that direction - perhaps Lars can offer more on that?).

Quoted 8 lines

GPIO is another interesting case - a lot of hardware is capable of parallel sampling, some at high speeds. It's another area that is probably too specialist for this discussion, but if people want to dive into the details it might be interesting.

I think we have only a small amount of fuzz around the v4l boundary, but wanted to leave the door open if anyone wants to discuss that one further as it's come up a few times over recent years.

Quoted 7 lines

Don't forget to take system integration into account. If I given you a high- speed ADC you will not think about V4L2 as your subsystem of choice. If the system designer has connected that ADC to a CPLD that generates fake horizontal and vertical sync signals, and connected the output to the camera interface of the SoC, you will be left with no choice but use the V4L2 API. That's largely a userspace issue in this case, but it implies that V4L2 need to define an "image format" for the ADC data.

This one was somewhat of a thought experiment (and the sort of dirty hack we'd do in my day job :)

Absolutely. Sometimes they'll obligingly connect it to a camera specific interface sometimes the won't. I think I saw an example of a camera hanging of one of TI's universal parallel ports at one point for example - not sure how the framing worked there.

The many generic high speed serial interfaces are also an option.

If it's all wrapped up and presented as a camera device, then we don't need to care that it's a generic ADC really as the interesting handling will probably all be done in the fpga / cpld anyway.

The possibility that gets more interesting from my point of view is where the pixel stream simply hits an ADC. Grabbing from the day job, a line scan camera would be the classic. There'd probably still be some sort of line framing but lots of places to hide that including in the analog stream itself.

Even if they do, there always comes a point where it's easier to just declare that we are better off with a monolithic device driver hiding what is inside.

Anyhow, I kind of hope no one ever does this *crosses fingers*

Possibly more likely around the blurry boundary of what is a video capture device and what is simply taking a small number of specially related signals and sampling them...

Quoted 26 lines
Quoted 25 lines

The SoC world is a major case of one device, many uses. Some SoCs are turning up with multiple ADC units, sometimes with different designs, sometimes simply so that the same hardware can be used for different things.

A few suggestions for people:

Guenter Roeck (hmwon)
Dmitry Torokhov (input)
Me (Jonathan Cameron - IIO)

Someone thermal related (not sure who would be most interested) Someone power supply related? (Guenter, any suggestions?)

Someone device tree related (Mark Rutland? Rob Herring?)

Linus Waleij (gpio)

Mark Brown - as some of concepts of IIO bridging and how far it can be taken (everything via a bridge driver) came from discussions with him on SoC ADC handling.

Lar-Peter Clausen for generally doing insane things with ADCs + having one foot in the userspace side of things.

#36 lore raw
Quoted 160 lines · Jonathan Cameron
Quoted 11 lines · Laurent Pinchart

Hi Jonathan,

Quoted 5 lines · On Wednesday 20 Jul 2016 22:18:11 Jonathan Cameron wrote:

Hi All,

This topic would be around the way the various subsystems interact, in the rough area of 'sensors' (I haven't yet had much of an issue with subsystem crossing with output devices but maybe that's just over the next hill!)

It's already here :-) We have an API to control camera flash (as in light, not memory) in V4L2 and another one in the LED subsystem.

I wonder how many devices are doing it with a gpio or some of the more configurable PWM units..

Quoted 66 lines
Quoted 57 lines

Scope may well be wider but includes:
* input (some of it)
* hwmon
* iio
* comedi(?)
* thermal
* power/battery
* gpio - the blur between gpios and beaglescope / PLC type I/O.
* v4l - when does a device jump from being a multi pixel thermopile to a thermal camera? Smart fingerprint scanners etc. Gesture recognition sensors (effectively 9ish pixel cameras)
* Lots of random things we haven't seen yet.

All these make use of devices that are at their hearts ADCs. Whilst it would nice if the underlying devices always fell into one clear category that is not always the case. Hardware descriptions (e.g. device tree) are an important area of contention.

The examples that follow are mostly based around IIO interactions with hwmon / input as that's my area of expertise. However, I know that similar issues occur at other boundaries.

Guenter Roeck has been working on hwmon / thermal interfacing recently. We also regularly push drivers one way or the other around that boundary.

Dmitry and I often have to decide on the IIO / input scope boundary (and we've moved it over time) + a number of cross over drivers have turned up recently.

The other subsystems I encounter have been 'quieter' in their interactions with me but many of the issues below apply to them as well.

A rough list of related topics that might be worth face to face discussion includes:

1) Are we happy with the somewhat adhoc divisions of subsystems as they stand?

I am personally happy enough with this, but it's worth checking everyone else is. This probably causes more pain for new submitters than it does for old hands. Perhaps we can come up with a checklist style doc to save people having to ask.

We have had quite a few cases where a whole driver is submitted to the 'wrong' subsystem. Not the best first interaction with the kernel world for new contributors. (This sort of area is attracts newbies as it's relatively simple and the hardware is fairly cheap and more than once their first experience with review as exactly this!)

If we aren't happy, what do we do about it?

I'm hoping we are happy or, at least, resigned to the current state of affairs, but think the question should be asked from time to time as the answer may well change + the discussion will highlight pinch points we can work on.

Quoted 8 lines

2) Bridging drivers - there will always be cross over cases. E.g. Generic ADCs used for battery voltage monitoring. We have a number of bridges in mainline already and others out of tree. How important are these and what features would make them more useful?

Is it ever sensible to have two drivers for the same part because the use cases are so different? (I hope there is always a better way but maybe not)

Quoted 6 lines

I'd love to say no, but we have already hit that problem. There are two drivers for the ADV7511 HDMI encoder, one in the media subsystem for V4L2 output devices (drivers/media/i2c/adv7511.c), and one in the DRM/KMS subsystem for display devices (drivers/gpu/drm/i2c/adv7511.c, being moved to drivers/gpu/drm/bridge/). Try compiling them both in the kernel and see how they race to probe the same device (enjoy the show, popcorns not included).

Nasty.

Quoted 66 lines
Quoted 57 lines

This covers both generic bridges (e.g. iio-hwmon) and device specific bridges (a good example of a touch screen driver turned up in my inbox yesterday).

3) Bindings (device tree and similar). When it is appropriate to use a device tree to describe the overlap (bridging of channels)? Sometimes there is obvious real hardware involved (a thermistor on a generic ADC input for example), but there are many grey areas.

Finally bashing out an answer to the issue device tree maintainers have with the iio-hwmon bindings would be great.
On a less biased note, that example is pretty fiddly to define but would act as a good basis to work from more generally.

We know we got it wrong (back in the dark ages), but it isn't obvious how to get it right!

IIO-input has been partly held outside mainline for years by this issue.

Also some practical design decisions to make around how to implement these mappings to work best with deferred probing etc.

4) Should we drop all the bindings for bridging between such subsystems and do it all from userspace?

I think we may end up with a hybrid of the two, but need to be able to make it work in 'standard' cases without userspace being involved. That hybrid solution may well be devicetree overlay based... Unclear so far.
+ plenty of crazy things that 'might' make sense where there isn't even the pretence of representing real hardware.

5) Complex device interaction usecases. At the moment the ones I've come across are mostly contained in IIO. The moment we start sticking in MUXes, AFEs (Analog Front Ends) and straightforward analog sensors in the mix it can get fiddly. Swapping war stories may well be worthwhile on this. This stuff also turns up in ASoC for example so probably lessons to be learned from there.

The analog devices software defined radios are another possible case study.

There of course may well be lessons to be learned from similar interactions elsewhere in the kernel.

There is a lot of history in how we ended up where we are (it all made absolute sense at the time). Sitting back and taking the time to discuss the future would be great. Whilst this might be solvable by email we've made no definitive progress for years
(and what has been made has been on a case by case basis deep in driver reviews.)

I threw comedi in the list above but, at the moment, I think the more likely direction there is a single userspace library abstracting the underlying subsystem (Analog Devices are working in that direction - perhaps Lars can offer more on that?).

Quoted 8 lines

GPIO is another interesting case - a lot of hardware is capable of parallel sampling, some at high speeds. It's another area that is probably too specialist for this discussion, but if people want to dive into the details it might be interesting.

I think we have only a small amount of fuzz around the v4l boundary, but wanted to leave the door open if anyone wants to discuss that one further as it's come up a few times over recent years.

Quoted 7 lines

Don't forget to take system integration into account. If I given you a high- speed ADC you will not think about V4L2 as your subsystem of choice. If the system designer has connected that ADC to a CPLD that generates fake horizontal and vertical sync signals, and connected the output to the camera interface of the SoC, you will be left with no choice but use the V4L2 API. That's largely a userspace issue in this case, but it implies that V4L2 need to define an "image format" for the ADC data.

Quoted 7 lines

This one was somewhat of a thought experiment (and the sort of dirty hack we'd do in my day job :)

Absolutely. Sometimes they'll obligingly connect it to a camera specific interface sometimes the won't. I think I saw an example of a camera hanging of one of TI's universal parallel ports at one point for example - not sure how the framing worked there.

One of the first v4l drivers was for a parallel port webcam.

And I just heard of a device that uses spi for the video transfer. HW designers are evil...

Regards,

Hans

Quoted 54 lines

The many generic high speed serial interfaces are also an option.

If it's all wrapped up and presented as a camera device, then we don't need to care that it's a generic ADC really as the interesting handling will probably all be done in the fpga / cpld anyway.

The possibility that gets more interesting from my point of view is where the pixel stream simply hits an ADC. Grabbing from the day job, a line scan camera would be the classic. There'd probably still be some sort of line framing but lots of places to hide that including in the analog stream itself.

Even if they do, there always comes a point where it's easier to just declare that we are better off with a monolithic device driver hiding what is inside.

Anyhow, I kind of hope no one ever does this *crosses fingers*

Possibly more likely around the blurry boundary of what is a video capture device and what is simply taking a small number of specially related signals and sampling them...

Quoted 17 lines
Quoted 17 lines

The SoC world is a major case of one device, many uses. Some SoCs are turning up with multiple ADC units, sometimes with different designs, sometimes simply so that the same hardware can be used for different things.

A few suggestions for people:

Guenter Roeck (hmwon)
Dmitry Torokhov (input)
Me (Jonathan Cameron - IIO)

Someone thermal related (not sure who would be most interested) Someone power supply related? (Guenter, any suggestions?)

Someone device tree related (Mark Rutland? Rob Herring?)

Quoted 9 lines
Quoted 8 lines

Linus Waleij (gpio)

Mark Brown - as some of concepts of IIO bridging and how far it can be taken (everything via a bridge driver) came from discussions with him on SoC ADC handling.

Lar-Peter Clausen for generally doing insane things with ADCs + having one foot in the userspace side of things.

_______________________________________________
Ksummit-discuss mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

#37 lore raw
In reply to Jonathan Cameron

Hi Jonathan,

Quoted 9 lines · On Thursday 28 Jul 2016 23:26:26 Jonathan Cameron wrote:
Quoted 8 lines · Laurent Pinchart
Quoted 6 lines · On Wednesday 20 Jul 2016 22:18:11 Jonathan Cameron wrote:

Hi All,

This topic would be around the way the various subsystems interact, in the rough area of 'sensors' (I haven't yet had much of an issue with subsystem crossing with output devices but maybe that's just over the next hill!)

Quoted 5 lines
Quoted 2 lines

It's already here :-) We have an API to control camera flash (as in light, not memory) in V4L2 and another one in the LED subsystem.

I wonder how many devices are doing it with a gpio or some of the more configurable PWM units..

At least on phone platforms the device usually has a dedicated chip for this, as there are safety implications that require a hardware timeout (at the highest current leaving the LED flash always on could damage the LED, cause the phone to overhead and catch fire, and possibly even damage your eyes).

Quoted 149 lines
Quoted 65 lines
Quoted 56 lines

Scope may well be wider but includes:
* input (some of it)
* hwmon
* iio
* comedi(?)
* thermal
* power/battery
* gpio - the blur between gpios and beaglescope / PLC type I/O.
* v4l - when does a device jump from being a multi pixel thermopile to a thermal camera? Smart fingerprint scanners etc. Gesture recognition sensors (effectively 9ish pixel cameras)
* Lots of random things we haven't seen yet.

All these make use of devices that are at their hearts ADCs. Whilst it would nice if the underlying devices always fell into one clear category that is not always the case. Hardware descriptions (e.g. device tree) are an important area of contention.

The examples that follow are mostly based around IIO interactions with hwmon / input as that's my area of expertise. However, I know that similar issues occur at other boundaries.

Guenter Roeck has been working on hwmon / thermal interfacing recently. We also regularly push drivers one way or the other around that boundary.

Dmitry and I often have to decide on the IIO / input scope boundary (and we've moved it over time) + a number of cross over drivers have turned up recently.

The other subsystems I encounter have been 'quieter' in their interactions with me but many of the issues below apply to them as well.

A rough list of related topics that might be worth face to face discussion includes:

1) Are we happy with the somewhat adhoc divisions of subsystems as they stand?

I am personally happy enough with this, but it's worth checking everyone else is. This probably causes more pain for new submitters than it does for old hands. Perhaps we can come up with a checklist style doc to save people having to ask.

We have had quite a few cases where a whole driver is submitted to the 'wrong' subsystem. Not the best first interaction with the kernel world for new contributors. (This sort of area is attracts newbies as it's relatively simple and the hardware is fairly cheap and more than once their first experience with review as exactly this!)

If we aren't happy, what do we do about it?

I'm hoping we are happy or, at least, resigned to the current state of affairs, but think the question should be asked from time to time as the answer may well change + the discussion will highlight pinch points we can work on.

Quoted 8 lines

2) Bridging drivers - there will always be cross over cases. E.g. Generic ADCs used for battery voltage monitoring. We have a number of bridges in mainline already and others out of tree. How important are these and what features would make them more useful?

Is it ever sensible to have two drivers for the same part because the use cases are so different? (I hope there is always a better way but maybe not)

Quoted 7 lines

I'd love to say no, but we have already hit that problem. There are two drivers for the ADV7511 HDMI encoder, one in the media subsystem for V4L2 output devices (drivers/media/i2c/adv7511.c), and one in the DRM/KMS subsystem for display devices (drivers/gpu/drm/i2c/adv7511.c, being moved to drivers/gpu/drm/bridge/). Try compiling them both in the kernel and see how they race to probe the same device (enjoy the show, popcorns not included).

Nasty.

Quoted 65 lines
Quoted 56 lines

This covers both generic bridges (e.g. iio-hwmon) and device specific bridges (a good example of a touch screen driver turned up in my inbox yesterday).

3) Bindings (device tree and similar). When it is appropriate to use a device tree to describe the overlap (bridging of channels)? Sometimes there is obvious real hardware involved (a thermistor on a generic ADC input for example), but there are many grey areas.

Finally bashing out an answer to the issue device tree maintainers have with the iio-hwmon bindings would be great.
On a less biased note, that example is pretty fiddly to define but would act as a good basis to work from more generally.

We know we got it wrong (back in the dark ages), but it isn't obvious how to get it right!

IIO-input has been partly held outside mainline for years by this issue.

Also some practical design decisions to make around how to implement these mappings to work best with deferred probing etc.

4) Should we drop all the bindings for bridging between such subsystems and do it all from userspace?

I think we may end up with a hybrid of the two, but need to be able to make it work in 'standard' cases without userspace being involved. That hybrid solution may well be devicetree overlay based... Unclear so far.
+ plenty of crazy things that 'might' make sense where there isn't even the pretence of representing real hardware.

5) Complex device interaction usecases. At the moment the ones I've come across are mostly contained in IIO. The moment we start sticking in MUXes, AFEs (Analog Front Ends) and straightforward analog sensors in the mix it can get fiddly. Swapping war stories may well be worthwhile on this. This stuff also turns up in ASoC for example so probably lessons to be learned from there.

The analog devices software defined radios are another possible case study.

There of course may well be lessons to be learned from similar interactions elsewhere in the kernel.

There is a lot of history in how we ended up where we are (it all made absolute sense at the time). Sitting back and taking the time to discuss the future would be great. Whilst this might be solvable by email we've made no definitive progress for years
(and what has been made has been on a case by case basis deep in driver reviews.)

I threw comedi in the list above but, at the moment, I think the more likely direction there is a single userspace library abstracting the underlying subsystem (Analog Devices are working in that direction - perhaps Lars can offer more on that?).

Quoted 8 lines

GPIO is another interesting case - a lot of hardware is capable of parallel sampling, some at high speeds. It's another area that is probably too specialist for this discussion, but if people want to dive into the details it might be interesting.

I think we have only a small amount of fuzz around the v4l boundary, but wanted to leave the door open if anyone wants to discuss that one further as it's come up a few times over recent years.

Quoted 8 lines

Don't forget to take system integration into account. If I given you a high-speed ADC you will not think about V4L2 as your subsystem of choice. If the system designer has connected that ADC to a CPLD that generates fake horizontal and vertical sync signals, and connected the output to the camera interface of the SoC, you will be left with no choice but use the V4L2 API. That's largely a userspace issue in this case, but it implies that V4L2 need to define an "image format" for the ADC data.

Quoted 2 lines

This one was somewhat of a thought experiment (and the sort of dirty hack we'd do in my day job :)

It's a use case a customer of mine implemented in a real product :-)

Quoted 51 lines

Absolutely. Sometimes they'll obligingly connect it to a camera specific interface sometimes the won't. I think I saw an example of a camera hanging of one of TI's universal parallel ports at one point for example - not sure how the framing worked there.

The many generic high speed serial interfaces are also an option.

If it's all wrapped up and presented as a camera device, then we don't need to care that it's a generic ADC really as the interesting handling will probably all be done in the fpga / cpld anyway.

The possibility that gets more interesting from my point of view is where the pixel stream simply hits an ADC. Grabbing from the day job, a line scan camera would be the classic. There'd probably still be some sort of line framing but lots of places to hide that including in the analog stream itself.

Even if they do, there always comes a point where it's easier to just declare that we are better off with a monolithic device driver hiding what is inside.

Anyhow, I kind of hope no one ever does this *crosses fingers*

Possibly more likely around the blurry boundary of what is a video capture device and what is simply taking a small number of specially related signals and sampling them...

Quoted 24 lines
Quoted 24 lines

The SoC world is a major case of one device, many uses. Some SoCs are turning up with multiple ADC units, sometimes with different designs, sometimes simply so that the same hardware can be used for different things.

A few suggestions for people:

Guenter Roeck (hmwon)
Dmitry Torokhov (input)
Me (Jonathan Cameron - IIO)

Someone thermal related (not sure who would be most interested) Someone power supply related? (Guenter, any suggestions?)

Someone device tree related (Mark Rutland? Rob Herring?)

Linus Waleij (gpio)

Mark Brown - as some of concepts of IIO bridging and how far it can be taken (everything via a bridge driver) came from discussions with him on SoC ADC handling.

Lar-Peter Clausen for generally doing insane things with ADCs + having one foot in the userspace side of things.

Signature · 3 lines
Regards,

Laurent Pinchart
#38 lore raw
In reply to Jonathan Cameron
Quoted 5 lines · Jonathan Cameron

Hi All,

This topic would be around the way the various subsystems interact, in the rough area of 'sensors' (I haven't yet had much of an issue with subsystem crossing with output devices but maybe that's just over the next hill!)

I've been on the other side of that hill and I can tell you there is also some output overlap. E.g. between regulator drivers and IIO DAC drivers. There are regulator drivers for general-purpose DACs that are no any different than other DACs supported by IIO. E.g. ad5398. Other overlaps might be backlight brightness control or lens control. In those cases you have some kind of hardware aperture connected to the output and the application is no longer ambiguous. But we should still avoid writing application specific drivers for this case and rather make sure that we have proper in-kernel frameworks so a generic driver can be controlled through the application specific userspace API.

#39 lore raw
In reply to Jonathan Cameron

Hi,

Quoted 11 lines · Jonathan Cameron

A few suggestions for people:

Guenter Roeck (hmwon)
Dmitry Torokhov (input)
Me (Jonathan Cameron - IIO)

Someone thermal related (not sure who would be most interested) Someone power supply related? (Guenter, any suggestions?)

Someone device tree related (Mark Rutland? Rob Herring?)

Quoted 9 lines

Linus Waleij (gpio)

Mark Brown - as some of concepts of IIO bridging and how far it can be taken (everything via a bridge driver) came from discussions with him on SoC ADC handling.

Lar-Peter Clausen for generally doing insane things with ADCs + having one foot in the userspace side of things.

I'm pretty interested in that discussion as I've been writing/using some of the ADC/touch drivers (at91, mxs) and I've been involved in the sunxi ones.

Also, I've recently been seeing people using GP ADCs wanting to get an ALSA interface and others using audio ADCs wanting the IIO interface.

Regards,

Signature · 3 lines
Alexandre Belloni, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
#40 lore raw
Quoted 25 lines · Alexandre Belloni

Hi,

Quoted 11 lines · Jonathan Cameron

A few suggestions for people:

Guenter Roeck (hmwon)
Dmitry Torokhov (input)
Me (Jonathan Cameron - IIO)

Someone thermal related (not sure who would be most interested) Someone power supply related? (Guenter, any suggestions?)

Someone device tree related (Mark Rutland? Rob Herring?)

Quoted 7 lines

Linus Waleij (gpio)

Mark Brown - as some of concepts of IIO bridging and how far it can be taken (everything via a bridge driver) came from discussions with him on SoC ADC handling.

Lar-Peter Clausen for generally doing insane things with ADCs +

having

Quoted 2 lines

one foot in the userspace side of things.

Quoted 8 lines

I'm pretty interested in that discussion as I've been writing/using some
of the ADC/touch drivers (at91, mxs) and I've been involved in the sunxi
ones.

Also, I've recently been seeing people using GP ADCs wanting to get an ALSA interface and others using audio ADCs wanting the IIO interface.

So here is another case where it looks like we ultimately need bridges.

Hopefully the hardware usecase is obvious enough here that bindings will be straight forward anyway...

Fits in well with Lars' point about everything is going more generic and we need to be ready for it. Sounds like we are a bit late already...

I am beginning to wonder why I opened this can of worms!

Jonathan

Quoted 2 lines

Regards,

Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.