is it possible to use single IOCTL to setup media pipeline?
Hello everyone
when we need to setup media pipeline, eg, for camera capture, media-ctl needs to be called multiple time to setup media link and subdev formats, or similar code in a single application. this will use multiple IOCTLs on "/dev/mediaX" and "/dev/v4l2-subdevY".
to setup media pipeline in userspace requires to fully understanding the topology of the media stack. but the fact is only media driver developer could know how to setup media pipeline. each time driver updates, this would break userspace application if application engineers don't know this change. In this case, if a IOCTL is designed to setup media pipeline, no need to update applications, after driver is updated.
this will not only benefit for design a single IOCTL, this also helps to hide the detail of media pipeline, by load a binary blob which holds information about how to setup pipeline, or hide it in bootloader/ACPI tables/device tree, etc.
another benefit is save time for setup media pipeline, if there is a PKI like "time for open camera". as my test, this will saves hundreds of milliseconds.
is this acceptable?
BR.
Ning.
Hi Ning,
Quoted 8 lines · Ning A
Hello everyone
when we need to setup media pipeline, eg, for camera capture, media-ctl needs to be called multiple time to setup media link and subdev formats, or similar code in a single application. this will use multiple IOCTLs on "
/dev/mediaX" and "/dev/v4l2-subdevY".
Quoted 5 lines
to setup media pipeline in userspace requires to fully understanding the topology of the media stack. but the fact is only media driver developer could know how to setup media pipeline. each time driver updates, this would break userspace application if application engineers don't know this change.
That's obviously a bug in the driver. Kernel interfaces must not change in a way that are not compatible with the userspace.
Quoted 8 lines
In this case, if a IOCTL is designed
to setup media pipeline, no need to update applications, after driver is updated.this will not only benefit for design a single IOCTL, this also helps to hide the detail of media pipeline, by load a binary blob which holds information about how to setup pipeline, or hide it in bootloader/ACPI tables/device tree, etc.
Media pipeline configuration is specific to the use case. If you hardcode it in the driver or bootloader, the user will not be able to use any other use case than the hardcoded blob, which is unacceptable for Linux drivers.
Instead, it sounds like your userspace should be designed in a way that the media topology configuration is loaded from a configuration file that you could either get from your kernel driver developer or just maintain yourself based on any changes the media developers do. Of course that's unrelated to the backwards compatibility issue, which should not happen normally. The configuration file would be helpful for handling future extensions and new hardware platforms.
Quoted 4 lines
another benefit is save time for setup media pipeline, if there is a PKI like "time for open camera". as my test, this will saves hundreds of milliseconds.
For this problem, the proper solution would be to create an ioctl that can aggregate setting multiple parts of the topology in one go. For example, V4L2 has VIDIOC_S_CTRL for setting a control, but there is also VIDIOC_S_EXT_CTRLS, which lets you set multiple controls in one call. Something like VIDIOC_S_EXT_CTRLS for configuring the media topology would solve the performance problem.
Best regards,
Tomasz
Tomasz, Thank you for your answers.
learned a lot.
BR.
Ning.
Quoted 20 lines · 在 2018-11-22四的 16:06 +0900，Tomasz Figa写道：
Hi Ning,
Quoted 9 lines · Ning A
Hello everyone
when we need to setup media pipeline, eg, for camera capture, media-ctl
needs to be called multiple time to setup media link and subdev formats, or similar code in a single application. this will use multiple IOCTLs on "/dev/mediaX" and "/dev/v4l2-subdevY".Quoted 6 lines
to setup media pipeline in userspace requires to fully understanding
the topology of the media stack. but the fact is only media driver developer could know how to setup media pipeline. each time driver updates, this would break userspace application if application engineers don't know this change.
Quoted 2 lines
That's obviously a bug in the driver. Kernel interfaces must not change in a way that are not compatible with the userspace.
I met this issue :(
Quoted 19 lines
Quoted 6 lines
In this case, if a IOCTL is designed
to setup media pipeline, no need to update applications, after driver
is updated.Quoted 7 lines
this will not only benefit for design a single IOCTL, this also helps
to hide the detail of media pipeline, by load a binary blob which holds
information about how to setup pipeline, or hide it in bootloader/ACPI
tables/device tree, etc.Media pipeline configuration is specific to the use case. If you hardcode it in the driver or bootloader, the user will not be able to use any other use case than the hardcoded blob, which is unacceptable for Linux drivers.
Quoted 8 lines
Instead, it sounds like your userspace should be designed in a way that the media topology configuration is loaded from a configuration file that you could either get from your kernel driver developer or just maintain yourself based on any changes the media developers do. Of course that's unrelated to the backwards compatibility issue, which
should not happen normally. The configuration file would be helpful for handling future extensions and new hardware platforms.
yes, if there are multiple user cases, then this is not optional.
Quoted 8 lines
Quoted 7 lines
another benefit is save time for setup media pipeline, if there is a
PKI like "time for open camera". as my test, this will saves hundreds
of milliseconds.
Quoted 7 lines
For this problem, the proper solution would be to create an ioctl that
can aggregate setting multiple parts of the topology in one go. For example, V4L2 hasVIDIOC_S_CTRLfor setting a control, but there is alsoVIDIOC_S_EXT_CTRLS, which lets you set multiple controls in one call. Something likeVIDIOC_S_EXT_CTRLSfor configuring the media topology would solve the performance problem.
this may be alternative choice.
Quoted 4 lines
Best regards,
Tomasz
Hi Tomasz,
Quoted 11 lines · Tomasz Figa
Hi Ning,
Quoted 8 lines · Ning A
Hello everyone
when we need to setup media pipeline, eg, for camera capture, media-ctl needs to be called multiple time to setup media link and subdev formats, or similar code in a single application. this will use multiple IOCTLs on "
/dev/mediaX" and "/dev/v4l2-subdevY".
Quoted 8 lines
Quoted 5 lines
to setup media pipeline in userspace requires to fully understanding the topology of the media stack. but the fact is only media driver developer could know how to setup media pipeline. each time driver updates, this would break userspace application if application engineers don't know this change.
That's obviously a bug in the driver. Kernel interfaces must not change in a way that are not compatible with the userspace.
Alternatively, this could result from fixing a bug in a driver. Or adding features that were not previously supported.
In this case there are often no perfect solutions to address all the issues at hand.
Quoted 28 lines
Quoted 9 lines
In this case, if a IOCTL is designed
to setup media pipeline, no need to update applications, after driver is updated.this will not only benefit for design a single IOCTL, this also helps to hide the detail of media pipeline, by load a binary blob which holds information about how to setup pipeline, or hide it in bootloader/ACPI tables/device tree, etc.
Media pipeline configuration is specific to the use case. If you hardcode it in the driver or bootloader, the user will not be able to use any other use case than the hardcoded blob, which is unacceptable for Linux drivers.
Instead, it sounds like your userspace should be designed in a way that the media topology configuration is loaded from a configuration file that you could either get from your kernel driver developer or just maintain yourself based on any changes the media developers do. Of course that's unrelated to the backwards compatibility issue, which should not happen normally. The configuration file would be helpful for handling future extensions and new hardware platforms.
Quoted 4 lines
another benefit is save time for setup media pipeline, if there is a PKI like "time for open camera". as my test, this will saves hundreds of milliseconds.
Quoted 6 lines
For this problem, the proper solution would be to create an ioctl that can aggregate setting multiple parts of the topology in one go. For example, V4L2 has
VIDIOC_S_CTRLfor setting a control, but there is alsoVIDIOC_S_EXT_CTRLS, which lets you set multiple controls in one call. Something likeVIDIOC_S_EXT_CTRLSfor configuring the media topology would solve the performance problem.
There have been ideas of moving all IOCTL handling to the media device, in a way that would allow issuing them in the same fashion than controls. This was discussed in last year's media summit actually. I think this could be done once the request API is otherwise working for e.g. camera devices. I don't expect this to materialise in near (or even medium) term though.
Signature · 4 lines
Regards, Sakari Ailus e-mail: [email protected]
Quoted 21 lines · Sakari Ailus
Hi Tomasz,
Quoted 17 lines · Tomasz Figa
Hi Ning,
Quoted 13 lines · Ning A
Hello everyone
when we need to setup media pipeline, eg, for camera capture, media-ctl needs to be called multiple time to setup media link and subdev formats, or similar code in a single application. this will use multiple IOCTLs on "
/dev/mediaX" and "/dev/v4l2-subdevY".to setup media pipeline in userspace requires to fully understanding the topology of the media stack. but the fact is only media driver developer could know how to setup media pipeline. each time driver updates, this would break userspace application if application engineers don't know this change.
Quoted 8 lines
Quoted 2 lines
That's obviously a bug in the driver. Kernel interfaces must not change in a way that are not compatible with the userspace.
Alternatively, this could result from fixing a bug in a driver. Or adding features that were not previously supported.
In this case there are often no perfect solutions to address all the issues at hand.
An upstream kernel driver must maintain compatibility even in such cases, which isn't the most convenient thing for driver developers, but that's something we have to live with if we want to have users.
Quoted 36 lines
Quoted 29 lines
Quoted 10 lines
In this case, if a IOCTL is designed
to setup media pipeline, no need to update applications, after driver is updated.this will not only benefit for design a single IOCTL, this also helps to hide the detail of media pipeline, by load a binary blob which holds information about how to setup pipeline, or hide it in bootloader/ACPI tables/device tree, etc.
Media pipeline configuration is specific to the use case. If you hardcode it in the driver or bootloader, the user will not be able to use any other use case than the hardcoded blob, which is unacceptable for Linux drivers.
Instead, it sounds like your userspace should be designed in a way that the media topology configuration is loaded from a configuration file that you could either get from your kernel driver developer or just maintain yourself based on any changes the media developers do. Of course that's unrelated to the backwards compatibility issue, which should not happen normally. The configuration file would be helpful for handling future extensions and new hardware platforms.
Quoted 4 lines
another benefit is save time for setup media pipeline, if there is a PKI like "time for open camera". as my test, this will saves hundreds of milliseconds.
Quoted 6 lines
For this problem, the proper solution would be to create an ioctl that can aggregate setting multiple parts of the topology in one go. For example, V4L2 has
VIDIOC_S_CTRLfor setting a control, but there is alsoVIDIOC_S_EXT_CTRLS, which lets you set multiple controls in one call. Something likeVIDIOC_S_EXT_CTRLSfor configuring the media topology would solve the performance problem.
Quoted 5 lines
There have been ideas of moving all IOCTL handling to the media device, in a way that would allow issuing them in the same fashion than controls. This was discussed in last year's media summit actually. I think this could be done once the request API is otherwise working for e.g. camera devices. I don't expect this to materialise in near (or even medium) term though.
I'm aware of those talks and we actually took this into consideration when developing the Request API.
Hopefully we can speed up the work on this to make it happen medium term at latest, if not short term, as it's going to be quite important for the complex camera subsystems, especially in the days of all the counter measures against the speculative execution vulnerabilities.
Best regards,
Tomasz