Media Controller
Along the years, we've been developing a mechanism at the Kernel to store and present graphs to userspace, called Media Controller. The original scope of the Media Controller were to store and represent pipeline connections between hardware components on complex streaming devices, like the ones that are used on cell phones, where there are typically two camera sensors, and lots of processing units to enhance the image and to convert it into different formats.
Since the beginning, we found that it would be useful to use it some day on other subsystems. Well, now such day has come ;)
Driven by the need of using the media controller on TV sets, where the pipelines can be really complex and may cover more than one subsystem, we're reworking at both the core implementation and at the userspace API to allow it to be used where needed.
On a TV set, the streaming data pipeline covers camera sensors, analog TV, digital TV, crypto/decrypto modules, audio and GPU. Also, TVs have network interfaces that may be provided via DVB (like CATV). If we want/need to represent and control the full pipeline, the media controller has to have support on different subsystems: V4L2, DVB, ALSA, DRM and Network. It also needs a way to share a common struct between those subsystems without making them depend on each other. So, it should use devres.
Also, on the MC summit, it was pointed that other subsystems like IIO (Linux Industrial I/O) have similar needs and also need to track complex pipelines.
When we have such complex pipelines that interact with different subsystems, we need to better describe the interfaces between userspace and the pipeline. In the past, we only needed to represent device nodes, but now we'll need to be capable of representing network interfaces and sysfs.
We also need to allow dynamic changes in terms of removing and adding objects at the graph.
Due to that, we're redesigning the MC core, making it more generic and more capable of representing complex devices. So, we'd like to discuss it with the KS audience, in order to check if there are other use cases for it, and if the solution we're designing would require more changes to fit such other use cases.
For this discussion, I nominate myself and Laurent Pinchart, Hans Verkuil, Sakari Ailus, Shuah Khan and Lars-Peter Clausen.
PS.: I know we're a week late, but we've been busy this month with the discussions, including a Media Controller 3-days summit in Helsinki last week.
Quoted 36 lines · Mauro Carvalho Chehab
Along the years, we've been developing a mechanism at the Kernel to store and present graphs to userspace, called Media Controller. The original scope of the Media Controller were to store and represent pipeline connections between hardware components on complex streaming devices, like the ones that are used on cell phones, where there are typically two camera sensors, and lots of processing units to enhance the image and to convert it into different formats.
Since the beginning, we found that it would be useful to use it some day on other subsystems. Well, now such day has come ;)
Driven by the need of using the media controller on TV sets, where the pipelines can be really complex and may cover more than one subsystem, we're reworking at both the core implementation and at the userspace API to allow it to be used where needed.
On a TV set, the streaming data pipeline covers camera sensors, analog TV, digital TV,
crypto/decryptomodules, audio and GPU. Also, TVs have network interfaces that may be provided via DVB (like CATV). If we want/need to represent and control the full pipeline, the media controller has to have support on different subsystems: V4L2, DVB, ALSA, DRM and Network. It also needs a way to share a common struct between those subsystems without making them depend on each other. So, it should use devres.Also, on the MC summit, it was pointed that other subsystems like IIO (Linux Industrial I/O) have similar needs and also need to track complex pipelines.
When we have such complex pipelines that interact with different subsystems, we need to better describe the interfaces between userspace and the pipeline. In the past, we only needed to represent device nodes, but now we'll need to be capable of representing network interfaces and sysfs.
We also need to allow dynamic changes in terms of removing and adding objects at the graph.
Quoted 8 lines
Due to that, we're redesigning the MC core, making it more generic and more capable of representing complex devices. So, we'd like to discuss it with the KS audience, in order to check if there are other use cases for it, and if the solution we're designing would require more changes to fit such other use cases.
For this discussion, I nominate myself and Laurent Pinchart, Hans Verkuil, Sakari Ailus, Shuah Khan and Lars-Peter Clausen.
There's ideas for live sources/sinks to connect kms/drm drivers with v4l drivers. I want to participate too to discuss possible integration issues with drm. Dave Airlie should probably be there too. -Daniel
Quoted 7 lines
PS.: I know we're a week late, but we've been busy this month with the discussions, including a Media Controller 3-days summit in Helsinki last week.
_______________________________________________
Ksummit-discuss mailing list [email protected]
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Quoted 2 lines · Mauro Carvalho Chehab
For this discussion, I nominate myself and Laurent Pinchart, Hans Verkuil, Sakari Ailus, Shuah Khan and Lars-Peter Clausen.
I'm interested in this as well, as you know it's something that's been discussed for years with ASoC though there's not been much active work (but Lars has said he's working on it at the minute so perhaps we'll see something soon).
Quoted 36 lines · Mauro Carvalho Chehab
Along the years, we've been developing a mechanism at the Kernel to store
and present graphs to userspace, called Media Controller. The original scope of the Media Controller were to store and represent pipeline connections
between hardware components on complex streaming devices, like the ones that are used on cell phones, where there are typically two camera sensors,
and lots of processing units to enhance the image and to convert it into
different formats.Since the beginning, we found that it would be useful to use it some day
on other subsystems. Well, now such day has come ;)Driven by the need of using the media controller on TV sets, where the pipelines can be really complex and may cover more than one subsystem, we're reworking at both the core implementation and at the userspace API
to allow it to be used where needed.On a TV set, the streaming data pipeline covers camera sensors, analog TV,
digital TV,crypto/decryptomodules, audio and GPU. Also, TVs have network
interfaces that may be provided via DVB (like CATV). If we want/need torepresent and control the full pipeline, the media controller has to have
support on different subsystems: V4L2, DVB, ALSA, DRM and Network. It also
needs a way to share a common struct between those subsystems without making
them depend on each other. So, it should use devres.
Quoted 4 lines
Also, on the MC summit, it was pointed that other subsystems like IIO (Linux Industrial I/O) have similar needs and also need to track complex
pipelines.
Not yet as complex as some of your media cases though I wouldn't be
surprised if Lars has stuff for this underway!
I need to catch up with your framework as I lost track a while back. Lars-Peter
Clausen is probably the best person to be present for the IIO elements of this (on top of other roles in this discussion!)
Thankfully most of the sensor hub stuff is probably still where media devices were a while back in that the inputs are hardwired to the DSP elements and
mostly hidden from the host processor.
Of course there are probably more complex topologies already out there
somewhere!
Jonathan
Quoted 29 lines
When we have such complex pipelines that interact with different subsystems,
we need to better describe the interfaces between userspace and the pipeline. In the past, we only needed to represent device nodes, but now
we'll need to be capable of representing network interfaces and sysfs.We also need to allow dynamic changes in terms of removing and adding objects at the graph.
Due to that, we're redesigning the MC core, making it more generic and more capable of representing complex devices. So, we'd like to discuss it with the KS audience, in order to check if there are other use cases for it, and if the solution we're designing would require more changes to fit such other use cases.
For this discussion, I nominate myself and Laurent Pinchart, Hans Verkuil,
Sakari Ailus, Shuah Khan and Lars-Peter Clausen.PS.: I know we're a week late, but we've been busy this month with the discussions, including a Media Controller 3-days summit in Helsinki last
week.
_______________________________________________
Ksummit-discuss mailing list [email protected]
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Quoted 55 lines · Jonathan Cameron
Quoted 6 lines · Mauro Carvalho Chehab
Along the years, we've been developing a mechanism at the Kernel to store
and present graphs to userspace, called Media Controller. The original scope of the Media Controller were to store and represent pipeline connections
between hardware components on complex streaming devices, like theones
Quoted 16 lines
that are used on cell phones, where there are typically two camera sensors,
and lots of processing units to enhance the image and to convert it into
different formats.Since the beginning, we found that it would be useful to use it some day
on other subsystems. Well, now such day has come ;)Driven by the need of using the media controller on TV sets, where the pipelines can be really complex and may cover more than one subsystem, we're reworking at both the core implementation and at the userspace API
to allow it to be used where needed.Quoted 5 lines
On a TV set, the streaming data pipeline covers camera sensors, analog TV,
digital TV,crypto/decryptomodules, audio and GPU. Also, TVs have network
interfaces that may be provided via DVB (like CATV). If we want/needto
Quoted 9 lines
represent and control the full pipeline, the media controller has to have
support on different subsystems: V4L2, DVB, ALSA, DRM and Network. It also
needs a way to share a common struct between those subsystems without making
them depend on each other. So, it should use devres.Quoted 4 lines
Also, on the MC summit, it was pointed that other subsystems like IIO (Linux Industrial I/O) have similar needs and also need to track complex
pipelines.Not yet as complex as some of your media cases though I wouldn't be surprised if Lars has stuff for this underway!
I need to catch up with your framework as I lost track a while back. Lars-Peter
Clausen is probably the best person to be present for the IIO elements of this (on top of other roles in this discussion!)
Quoted 6 lines
Thankfully most of the sensor hub stuff is probably still where media devices were a while back in that the inputs are hardwired to the DSP elements and
mostly hidden from the host processor.
Of course there are probably more complex topologies already out there somewhere!
Having had a bit of a read of the docs, I can see the framework could be used a lot more widely in IIO than I originally thought. We have a few different
dataflows:
* streamed main data flows (triggered samples from ADC channels) might be from an analogous accelerometer, through an ADC and on top a sink that outputs to userspace as an input driver.
* polled data flows (more like querying a regulator) not sure how these would be handled. Note some devices are very slow 1hz? Hence this path is very lightweight in terms of driver support needed. More or less the same as reading a hwmon channel from sysfs.
* trigger data flows. Hardware of software 'grab a sample now' signals. Kind of a one to many sync signal.
* event data flows (threshold detectors
etc - not yet handled well in IIO)
Can certainly see the streamed stuff and events fitting well with the media controller framework. Perhaps the triggering as well.
Any thoughts on the polled data flow?
Definitely an interesting topic to pursue!
Jonathan
Quoted 32 lines
Jonathan
Quoted 8 lines
When we have such complex pipelines that interact with different subsystems,
we need to better describe the interfaces between userspace and the pipeline. In the past, we only needed to represent device nodes, but now
we'll need to be capable of representing network interfaces and sysfs.Quoted 6 lines
We also need to allow dynamic changes in terms of removing and adding objects at the graph.
Due to that, we're redesigning the MC core, making it more generic and more capable of representing complex devices. So, we'd like to discuss it with the KS audience, in order to check if there are other use
cases
Quoted 15 lines
for it, and if the solution we're designing would require more changes to fit such other use cases.
For this discussion, I nominate myself and Laurent Pinchart, Hans Verkuil,
Sakari Ailus, Shuah Khan and Lars-Peter Clausen.PS.: I know we're a week late, but we've been busy this month with the discussions, including a Media Controller 3-days summit in Helsinki last
week.
_______________________________________________
Ksummit-discuss mailing list [email protected]
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Quoted 45 lines · Jonathan Cameron
Quoted 42 lines · Jonathan Cameron
Quoted 6 lines · Mauro Carvalho Chehab
Along the years, we've been developing a mechanism at the Kernel to store
and present graphs to userspace, called Media Controller. The original scope of the Media Controller were to store and represent pipeline connections
between hardware components on complex streaming devices, like theones
Quoted 16 lines
that are used on cell phones, where there are typically two camera sensors,
and lots of processing units to enhance the image and to convert it into
different formats.Since the beginning, we found that it would be useful to use it some day
on other subsystems. Well, now such day has come ;)Driven by the need of using the media controller on TV sets, where the pipelines can be really complex and may cover more than one subsystem, we're reworking at both the core implementation and at the userspace API
to allow it to be used where needed.Quoted 5 lines
On a TV set, the streaming data pipeline covers camera sensors, analog TV,
digital TV,crypto/decryptomodules, audio and GPU. Also, TVs have network
interfaces that may be provided via DVB (like CATV). If we want/needto
Quoted 9 lines
represent and control the full pipeline, the media controller has to have
support on different subsystems: V4L2, DVB, ALSA, DRM and Network. It also
needs a way to share a common struct between those subsystems without making
them depend on each other. So, it should use devres.
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Also, on the MC summit, it was pointed that other subsystems like IIO (Linux Industrial I/O) have similar needs and also need to track complex
pipelines.Not yet as complex as some of your media cases though I wouldn't be surprised if Lars has stuff for this underway!
This[1] is the quick presentation I gave the media controller workshop, it has a few examples of complex pipelines. Most of the driver support is work in progress, but it shows why we need a method of describing the data flow topology. So the goal is to have the MC integration ready when the drivers get merged.
- Lars
Quoted 51 lines · Lars-Peter Clausen
Quoted 50 lines · Jonathan Cameron
Quoted 47 lines · Jonathan Cameron
Quoted 6 lines · Mauro Carvalho Chehab
Along the years, we've been developing a mechanism at the Kernel to store
and present graphs to userspace, called Media Controller. The original scope of the Media Controller were to store and represent pipeline connections
between hardware components on complex streaming devices, like theones
Quoted 16 lines
that are used on cell phones, where there are typically two camera sensors,
and lots of processing units to enhance the image and to convert it into
different formats.Since the beginning, we found that it would be useful to use it some day
on other subsystems. Well, now such day has come ;)Driven by the need of using the media controller on TV sets, where the pipelines can be really complex and may cover more than one subsystem, we're reworking at both the core implementation and at the userspace API
to allow it to be used where needed.Quoted 5 lines
On a TV set, the streaming data pipeline covers camera sensors, analog TV,
digital TV,crypto/decryptomodules, audio and GPU. Also, TVs have network
interfaces that may be provided via DVB (like CATV). If we want/needto
Quoted 13 lines
represent and control the full pipeline, the media controller has to have
support on different subsystems: V4L2, DVB, ALSA, DRM and Network. It also
needs a way to share a common struct between those subsystems without making
them depend on each other. So, it should use devres.Also, on the MC summit, it was pointed that other subsystems like IIO (Linux Industrial I/O) have similar needs and also need to track complex
pipelines.
Quoted 9 lines
Quoted 2 lines
Quoted 2 lines
Not yet as complex as some of your media cases though I wouldn't be surprised if Lars has stuff for this underway!
This[1] is the quick presentation I gave the media controller workshop, it has a few examples of complex pipelines. Most of the driver support is work in progress, but it shows why we need a method of describing the data flow topology. So the goal is to have the MC integration ready when the drivers get merged.
Sorry if you are seeing this meesage twice. My previous message didn't make to the KS because it was too large.
I have been working on updating snd-usb-audio and a media driver (au0828) that uses snd-usb-audio as its audio driver to use Media Controller API to share resources. Sharing resources across these varied set of drivers required adding Managed Media Controller API and now I am building on top. Managed Media Controller API creates media_device as a device resource at the parent USB device
which is the root device for this driver hierarchy.
We reviewed Patch series v5 at the Media Summit and I am working on v6. Patch series v5 is available at
https://git.kernel.org/cgit/linux/kernel/git/shuah/linux.git/log/?h=media_controller
As go along and do this work, I am refining and adding new MC interfaces as needed and creating handlers to enable drivers to coordinate initialization as each driver adds new entities. Here is the link to media graph for
Hauppauge HVR950Q Hybrid USB TV stick that id driven by au0828 (DVB, Video), and snd-usb-audio drivers. The media device is s Managed device resource.
https://drive.google.com/file/d/0B0NIL0BQg-AlSFJPelltTm82UGM/view
I look forward to sharing and discussing this work and MC use-cases for other driver classes at the workshop.
thanks,
-- Shuah
Quoted 6 lines · Mauro Carvalho Chehab
Due to that, we're redesigning the MC core, making it more generic and more capable of representing complex devices. So, we'd like to discuss it with the KS audience, in order to check if there are other use cases for it, and if the solution we're designing would require more changes to fit such other use cases.
Quoted 6 lines
For this discussion, I nominate myself and Laurent Pinchart, Hans Verkuil, Sakari Ailus, Shuah Khan and Lars-Peter Clausen.
PS.: I know we're a week late, but we've been busy this month with the discussions, including a Media Controller 3-days summit in Helsinki last week.
Sorry for replying super duper late, I wanted to ensure that I have travel go ahead before asking :)
So I would like an invitation for this WS, at Intel we have similar issues with DSP graphs. We are using shiny new ASoC topology framework in our DSP to get topology information and not hard code the graphs, but have no way to tell that to usermode.
I did look at media controller in past and would love to discuss more. In audio, people have been talking about this and I would be spending time on this later this year
Signature · 1 line
~Vinod