v4l2loopback and v4l2-mem2mem framework
Hi there linux-media experts,
I am working on the out-of-tree v4l2loopback module[1]. My current goal is to implement the videobuf2 api. However, it appears that I must use a different vbuf2_queue for the OUTPUT and CAPTURE types. It appears that the mem2mem framework has the ability to do just this. I came to this conclusion while looking at the vim2m test module's code. While trying to modify the loopback module, I have added some code to try to call v4l2_m2m_init. You can see my very WIP attempt at code here[2].
I have installed the kernel-devel package on Fedora, which. Unfortunately, I get he following error messages when trying to compile:
<errors>
ERROR: modpost: "v4l2_m2m_release" [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined!
ERROR: modpost: "v4l2_m2m_init" [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined! </errors>
So, I looked into it, and Module.symvers and /proc/kallsyms don't have those symbols listed, so I think the Fedora kernel was not compiled with those symbols exported.
Okay, so next I tried to build a custom kernel that does export those symbols. I managed to get it done by enabling the vim2m (VIDEO_VIM2M config option) module. The new Module.symvers includes the symbols; and the compile succeeds, which brings me to my final act.
I need to expose those symbols without enabling the vim2m module since it is a testing module. I do not see a way to do that explicitly. The config option in question appears to be V4L2_MEM2MEM_DEV. And I haven't found a way to enable it without enabling something that implicitly selects it (which vim2m does). Do y'all have any suggestions for how to get these symbols exported properly when vim2m is not enabled?
Thanks,
Wren Turkal (wt)
[1] https://github.com/umlaeute/v4l2loopback
[2] https://github.com/wt/v4l2loopback/tree/m2m_try
Okay, so I have found how to enable the option to be manually selected. Would a patch like the following be acceptable? Or is there a better way to do this?
drivers/media/v4l2-core/Kconfig +0 -0
@@ -54,7 +54,7 @@ config V4L2_H264
# Used by drivers that need v4l2-mem2mem.ko
config V4L2_MEM2MEM_DEV
- tristate
+ tristate "V4L2 mem2mem framework support"
depends on VIDEOBUF2_CORE
# Used by LED subsystem flash drivers
Thanks,
wt
Quoted 46 lines · Wren Turkal
Hi there linux-media experts,
I am working on the out-of-tree v4l2loopback module[1]. My current goal is to implement the videobuf2 api. However, it appears that I must use a different
vbuf2_queuefor the OUTPUT and CAPTURE types. It appears that the mem2mem framework has the ability to do just this. I came to this conclusion while looking at the vim2m test module's code. While trying to modify the loopback module, I have added some code to try to callv4l2_m2m_init. You can see my very WIP attempt at code here[2].I have installed the kernel-devel package on Fedora, which. Unfortunately, I get he following error messages when trying to compile:
<errors>
ERROR: modpost: "v4l2_m2m_release" [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined!
ERROR: modpost: "v4l2_m2m_init" [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined! </errors>So, I looked into it, and Module.symvers and
/proc/kallsymsdon't have those symbols listed, so I think the Fedora kernel was not compiled with those symbols exported.Okay, so next I tried to build a custom kernel that does export those symbols. I managed to get it done by enabling the vim2m (
VIDEO_VIM2Mconfig option) module. The new Module.symvers includes the symbols; and the compile succeeds, which brings me to my final act.I need to expose those symbols without enabling the vim2m module since it is a testing module. I do not see a way to do that explicitly. The config option in question appears to be
V4L2_MEM2MEM_DEV. And I haven't found a way to enable it without enabling something that implicitly selects it (which vim2m does). Do y'all have any suggestions for how to get these symbols exported properly when vim2m is not enabled?Thanks,
Wren Turkal (wt)[1] https://github.com/umlaeute/v4l2loopback
[2] https://github.com/wt/v4l2loopback/tree/m2m_try
Hi Wren,
How about you submit the loopback driver for integration in mainline? We can review the driver, give feedback and help you get it merged.
Ezequiel
Quoted 74 lines · Wren Turkal
Okay, so I have found how to enable the option to be manually selected. Would a patch like the following be acceptable? Or is there a better way to do this?
drivers/media/v4l2-core/Kconfig +1
-1@@ -54,7 +54,7 @@ config V4L2_H264 # Used by drivers that need v4l2-mem2mem.ko config V4L2_MEM2MEM_DEV - tristate + tristate "V4L2 mem2mem framework support" depends on VIDEOBUF2_CORE # Used by LED subsystem flash driversThanks,
wtQuoted 48 lines · Wren Turkal
Hi there linux-media experts,
I am working on the out-of-tree v4l2loopback module[1]. My current goal is to implement the videobuf2 api. However, it appears that I must use a different
vbuf2_queuefor the OUTPUT and CAPTURE types. It appears that the mem2mem framework has the ability to do just this. I came to this conclusion while looking at the vim2m test module's code. While trying to modify the loopback module, I have added some code to try to callv4l2_m2m_init. You can see my very WIP attempt at code here[2].I have installed the kernel-devel package on Fedora, which. Unfortunately, I get he following error messages when trying to compile:
<errors>
ERROR: modpost: "v4l2_m2m_release" [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined!
ERROR: modpost: "v4l2_m2m_init" [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined! </errors>So, I looked into it, and Module.symvers and
/proc/kallsymsdon't have those symbols listed, so I think the Fedora kernel was not compiled with those symbols exported.Okay, so next I tried to build a custom kernel that does export those symbols. I managed to get it done by enabling the vim2m (
VIDEO_VIM2Mconfig option) module. The new Module.symvers includes the symbols; and the compile succeeds, which brings me to my final act.I need to expose those symbols without enabling the vim2m module since it is a testing module. I do not see a way to do that explicitly. The config option in question appears to be
V4L2_MEM2MEM_DEV. And I haven't found a way to enable it without enabling something that implicitly selects it (which vim2m does). Do y'all have any suggestions for how to get these symbols exported properly when vim2m is not enabled?Thanks,
Wren Turkal (wt)[1] https://github.com/umlaeute/v4l2loopback
[2] https://github.com/wt/v4l2loopback/tree/m2m_try
Em Tue, 2 Mar 2021 05:01:16 -0300
Ezequiel Garcia <[email protected]> escreveu:
Quoted 4 lines
Hi Wren,
How about you submit the loopback driver for integration in mainline? We can review the driver, give feedback and help you get it merged.
Please, don't top post.
The loopback driver is not something that could be merged. It was already submitted a few times.
One of the main issues is that userspace may sleep, and this could cause some bad effects and even hangups due to V4L2 core and VB2 mutex and spin locks.
Regards,
Mauro
Quoted 78 lines
Ezequiel
Quoted 74 lines · Wren Turkal
Okay, so I have found how to enable the option to be manually selected. Would a patch like the following be acceptable? Or is there a better way to do this?
drivers/media/v4l2-core/Kconfig +1
-1@@ -54,7 +54,7 @@ config V4L2_H264 # Used by drivers that need v4l2-mem2mem.ko config V4L2_MEM2MEM_DEV - tristate + tristate "V4L2 mem2mem framework support" depends on VIDEOBUF2_CORE # Used by LED subsystem flash driversThanks,
wtQuoted 48 lines · Wren Turkal
Hi there linux-media experts,
I am working on the out-of-tree v4l2loopback module[1]. My current goal is to implement the videobuf2 api. However, it appears that I must use a different
vbuf2_queuefor the OUTPUT and CAPTURE types. It appears that the mem2mem framework has the ability to do just this. I came to this conclusion while looking at the vim2m test module's code. While trying to modify the loopback module, I have added some code to try to callv4l2_m2m_init. You can see my very WIP attempt at code here[2].I have installed the kernel-devel package on Fedora, which. Unfortunately, I get he following error messages when trying to compile:
<errors>
ERROR: modpost: "v4l2_m2m_release" [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined!
ERROR: modpost: "v4l2_m2m_init" [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined! </errors>So, I looked into it, and Module.symvers and
/proc/kallsymsdon't have those symbols listed, so I think the Fedora kernel was not compiled with those symbols exported.Okay, so next I tried to build a custom kernel that does export those symbols. I managed to get it done by enabling the vim2m (
VIDEO_VIM2Mconfig option) module. The new Module.symvers includes the symbols; and the compile succeeds, which brings me to my final act.I need to expose those symbols without enabling the vim2m module since it is a testing module. I do not see a way to do that explicitly. The config option in question appears to be
V4L2_MEM2MEM_DEV. And I haven't found a way to enable it without enabling something that implicitly selects it (which vim2m does). Do y'all have any suggestions for how to get these symbols exported properly when vim2m is not enabled?Thanks,
Wren Turkal (wt)[1] https://github.com/umlaeute/v4l2loopback
[2] https://github.com/wt/v4l2loopback/tree/m2m_try
Thanks,
Mauro
Quoted 9 lines · Mauro Carvalho Chehab
Em Tue, 2 Mar 2021 05:01:16 -0300
Ezequiel Garcia <[email protected]> escreveu:Quoted 4 lines
Hi Wren,
How about you submit the loopback driver for integration in mainline? We can review the driver, give feedback and help you get it merged.
Quoted 9 lines
Please, don't top post.
The loopback driver is not something that could be merged. It was already submitted a few times.
One of the main issues is that userspace may sleep, and this could cause some bad effects and even hangups due to V4L2 core and VB2 mutex and spin locks.
Can you elaborate on this?
I always felt that it's a shame for something so popular as the loopback to be out-of-tree and unsupported.
Thanks,
Ezequiel
Quoted 1 line · Mauro Carvalho Chehab
Please, don't top post.
Sorry about the top post.
Quoted 2 lines
The loopback driver is not something that could be merged. It was already submitted a few times.
I'm willing to develop something new if you think that would be a better approach. I just want the functionality of the loopback so that OBS (or something else) can be used to compose a video scene for use by all the apps on Linux.
Quoted 3 lines
One of the main issues is that userspace may sleep, and this could cause some bad effects and even hangups due to V4L2 core and VB2 mutex and spin locks.
Is this something that could be fixed? If so, I am happy to try to help do that.
wt
A message that is not in the archive (sent off-list or not yet received): CAAEAJfCSfwxv5NyaH05Dfiw4wm6vGxL_ajoFup6r-GKx_VymJA@mail.gmail.com
A message that is not in the archive (sent off-list or not yet received): [email protected]
Okay, so I tried to write up a little bit of an RFC to organize my thoughts. I would love to get your thoughts:
RFC: A loopback device for the v4l2 subsystem.
# What is needed?
The common denominator for apps that use webcams in linux is the v4l2 subsystem. The v4l2 subsystem could benefit from a loopback device that allows one to compose custom video streams and present them in a normal way from a /dev/video* device.
# Why is it needed?
The v4l2 is the framework is the kernel interface used for supporting video output and video capture. This brings the webcam video to apps like Cheese or a web browser or OBS.
However, there is no way to customize the composition of the video stream before capture on these apps. Of course, apps like Cheese have filters, but this is not something that is composible with other apps. This makes it impossible to do something like the following:
* Composite a video scene with OBS.
* Use that composed video as my webcam for my Zoom call.
I think this would be cool, and I think that it would allow interesting things to be done with video.
# Prior Art
The v4l2loopback project[1] provides a Linux kernel module to do just this. This module provides a misc device node used to control the loopback video devices. In particular, the misc device is used to create and destroy loopback video devices. It's somewhat similar to the /dev/loop-control dev node that is used to control block loop devices.
There is also the vim2m test driver[2] in the kernel that appears to do some similar things. This module is a module that is used for testing the v4l2-mem2mem framework. This module has a more modern structure with a platform device that is used as the context for the other devices. It also uses the m2m framework that I think could benefit an in-kernel loopback implementation. It is also important to note that this module does not generally ship with distribution kernels (like in Fedora), likely due to the "test" designation.
# Possible paths forward
## Integrate v4l2loopback into the kernel
There is a lot to cleanup in the v4l2loopback code in order to get it in shape for mainline inclusion. Having said that, this also a somewhat tested implementation. There are even packages for using this project from Ubuntu.
## Start with vim2m
I think the vim2m.c could provide a really strong base on which to start a loopback implementation for the kernel. I see two options:
* Extend vim2m to include more loopback functionality, including dynamic creation/deletion of video loopback devices.
* Copy the vim2m code to a new module and develop there.
The extend option is attractive so that a new module mustn't be developed.
However, I am not sure that I want to complicate that module with options that would required for a proper loopback implementation.
# My thoughts on where to go next
I think that the vim2m module provides a much nicer starting point for an implementation of a mainline module. I think the the ideal solution would start by using the vim2m source as a template and developing a new module that incorporates the important functionality from v4l2loopback.
# Assumptions I have
I have looked into the code, and the m2m framework appears to provide a pair of vb2_queues that can be used for capture and output. I am assuming that the framework would be reasonable to use for a loopback implementation.
[1] https://github.com/umlaeute/v4l2loopback
[2] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drivers/media/test-drivers/vim2m.c?h=v5.11.2
Thanks,
wt
Quoted 148 lines · Wren Turkal
Ezequiel,
Submitting for upstream inclusion is a good idea, and the driver is licensed appropriately (gpl v2) for the kernel. I have asked the current maintainer for their thoughts. They are supportive of the idea, but they are in the middle of implementing an new feature that they would like to finish. I will work it up for inclusion when that work concludes since it's a pretty major change.
Having said that, does the patch I posted make sense? I think that being able to enable the m2m framework without having to build in a m2m driver explicitly makes good sense.
Also, thinking about this a little more, it might make more sense use use vim2m and v4l2loopback as inspiration and make something like loopback that looks more like vim2m.
Things I like about vim2m vs v4l2loopback right now:
* vim2m uses the m2m framework
* vim2m is better organized by using a platform device to organize the devices that created under the context of the platform device. v4l2lb uses a misc device and doesn't explicitly place devices under in the context of that device.
* The misc device created by v4l2lb is used to dynamically add and remove v4l2lb video devices. It It uses new IOCTLs to do that, which I wonder if there is a better way.
* vim2m exposes config via a mechanism that doesn't require a new tool. v4l2lb requires a new control binary because of the custom ioctls.So, I have a couple questions:
* Would it be better to use vim2m as a model to implement something like v4l2loopback?
* If I were to do that should I copy vim2m to a new module, or should vim2m be extended. FWIW, vim2m appears to have two things against it:
* vim2m is a test v4l2 driver. It would need to be changed to build as a non-test driver. This is a pretty easyKconfigchange and maybe moving files to a more appropriate place in the linux source tree.
* v4l2loopback does have a number of features that vim2m does not have. I will say that I think that adding most importantant functionality (new formats and dynamic creation of loopback video devices in my mind) is probably very approachable and may give us a more modern linux module code.Give the above, I am kinda leaning toward extending vim2m code. So, do you have an opinion on the following? Should it be a new module, or should it literally extend and improve vim2m?
wt
Quoted 87 lines · Ezequiel Garcia
Hi Wren,
How about you submit the loopback driver for integration in mainline?
We can review the driver, give feedback and help you get it merged.
Ezequiel
On Mon, Mar 1, 2021, 5:32 PM Wren Turkal <[email protected] <mailto:[email protected]>> wrote:
Okay, so I have found how to enable the option to be manually selected.
Would a patch like the following be acceptable? Or is there a better way to do this?diff --git a/drivers/media/v4l2-core/Kconfig
b/drivers/media/v4l2-core/Kconfig
indexbf49f83cb86f..25004137a982100644
--- a/drivers/media/v4l2-core/Kconfig
+++ b/drivers/media/v4l2-core/Kconfig
@@ -54,7 +54,7 @@ configV4L2_H264# Used by drivers that need v4l2-mem2mem.ko config
V4L2_MEM2MEM_DEV
- tristate
+ tristate "V4L2 mem2mem framework support" depends onVIDEOBUF2_CORE# Used by LED subsystem flash drivers
Thanks,
wtOn 2/28/21 2:07 PM, Wren Turkal wrote:
> Hi there linux-media experts,
>
> I am working on the out-of-tree v4l2loopback module[1]. My > current goal is to implement the videobuf2 api. However, > it appears that I must use a differentvbuf2_queuefor the > OUTPUT and CAPTURE types. It appears that the mem2mem > framework has the ability to do just this. I came to this > conclusion while looking at the vim2m test module's code. > While trying to modify the loopback module, I have added > some code to try to callv4l2_m2m_init. You can see my very > WIP attempt at code here[2].
>
> I have installed the kernel-devel package on Fedora, which. > Unfortunately, I get he following error messages when trying > to compile:
>
> <errors>
> ERROR: modpost: "v4l2_m2m_release"
> [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined! > ERROR: modpost: "v4l2_m2m_init"
> [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined! > </errors>
>
> So, I looked into it, and Module.symvers and/proc/kallsyms> don't have those symbols listed, so I think the Fedora kernel > was not compiled with those symbols exported.
>
> Okay, so next I tried to build a custom kernel that does > export those symbols. I managed to get it done by enabling > the vim2m (VIDEO_VIM2Mconfig option) module. The new > Module.symvers includes the symbols; and the compile succeeds, > which brings me to my final act.
>
> I need to expose those symbols without enabling the vim2m > module since it is a testing module. I do not see a way > to do that explicitly. The config option in question
> appears to beV4L2_MEM2MEM_DEV. And I haven't found a > way to enable it without enabling something that implicitly > selects it (which vim2m does). Do y'all have any suggestions > for how to get these symbols exported properly when vim2m is > not enabled?
>
>
> Thanks,
> Wren Turkal (wt)
>
> [1] https://github.com/umlaeute/v4l2loopback
<https://github.com/umlaeute/v4l2loopback>
> [2] https://github.com/wt/v4l2loopback/tree/m2m_try
<https://github.com/wt/v4l2loopback/tree/m2m_try>
Le mardi 02 mars 2021 à 14:08 -0800, Wren Turkal a écrit :
Quoted 37 lines
Okay, so I tried to write up a little bit of an RFC to organize my thoughts. I would love to get your thoughts:
RFC: A loopback device for the v4l2 subsystem.
# What is needed?
The common denominator for apps that use webcams in linux is the v4l2 subsystem. The v4l2 subsystem could benefit from a loopback device that allows one to compose custom video streams and present them in a normal way from a
/dev/video*device.# Why is it needed?
The v4l2 is the framework is the kernel interface used for supporting video output and video capture. This brings the webcam video to apps like Cheese or a web browser or OBS.
However, there is no way to customize the composition of the video stream before capture on these apps. Of course, apps like Cheese have filters, but this is not something that is composible with other apps. This makes it impossible to do something like the following:
* Composite a video scene with OBS.
* Use that composed video as my webcam for my Zoom call.I think this would be cool, and I think that it would allow interesting things to be done with video.
# Prior Art
The v4l2loopback project[1] provides a Linux kernel module to do just this. This module provides a misc device node used to control the loopback video devices. In particular, the misc device is used to create and destroy loopback video devices. It's somewhat similar to the
/dev/loop-controldev node that is used to control block loop devices.
Quoted 8 lines
There is also the vim2m test driver[2] in the kernel that appears to do some similar things. This module is a module that is used for testing the v4l2-mem2mem framework. This module has a more modern structure with a platform device that is used as the context for the other devices. It also uses the m2m framework that I think could benefit an in-kernel loopback implementation. It is also important to note that this module does not generally ship with distribution kernels (like in Fedora), likely due to the "test" designation.
No, since m2m are instance base, you cannot have two process operating on the same node (there is only 1 streaming state). You'd probably want to look at extending the VIMC driver instead. It's a virtual media controller, it's missing few bits to make loopback possible, but would be a good fit.
Note that vivid have a loopback capability, but it does not constraint the width/height/format and will produce a test pattern if you request something that differ from the source.
Quoted 9 lines
# Possible paths forward
## Integrate v4l2loopback into the kernel
There is a lot to cleanup in the v4l2loopback code in order to get it in shape for mainline inclusion. Having said that, this also a somewhat tested implementation. There are even packages for using this project from Ubuntu.
It is very buggy, it often return a queued buffer back, and in fact, if you try to use the queue too much it fails. Though all this would probably be fine if you port it to VB2.
Quoted 189 lines
## Start with vim2m
I think the
vim2m.ccould provide a really strong base on which to start a loopback implementation for the kernel. I see two options:* Extend vim2m to include more loopback functionality, including dynamic creation/deletion of video loopback devices.
* Copy the vim2m code to a new module and develop there.The extend option is attractive so that a new module mustn't be developed.
However, I am not sure that I want to complicate that module with options that would required for a proper loopback implementation.
# My thoughts on where to go next
I think that the vim2m module provides a much nicer starting point for an implementation of a mainline module. I think the the ideal solution would start by using the vim2m source as a template and developing a new module that incorporates the important functionality from v4l2loopback.
# Assumptions I have
I have looked into the code, and the m2m framework appears to provide a pair of
vb2_queuesthat can be used for capture and output. I am assuming that the framework would be reasonable to use for a loopback implementation.[1] https://github.com/umlaeute/v4l2loopback
[2]Thanks,
wtQuoted 149 lines · Wren Turkal
Ezequiel,
Submitting for upstream inclusion is a good idea, and the driver is licensed appropriately (gpl v2) for the kernel. I have asked the current maintainer for their thoughts. They are supportive of the idea, but they are in the middle of implementing an new feature that they would like to finish. I will work it up for inclusion when that work concludes since it's a pretty major change.
Having said that, does the patch I posted make sense? I think that being able to enable the m2m framework without having to build in a m2m driver explicitly makes good sense.
Also, thinking about this a little more, it might make more sense use use vim2m and v4l2loopback as inspiration and make something like loopback that looks more like vim2m.
Things I like about vim2m vs v4l2loopback right now:
* vim2m uses the m2m framework
* vim2m is better organized by using a platform device to organize the devices that created under the context of the platform device. v4l2lb uses a misc device and doesn't explicitly place devices under in the context of that device.
* The misc device created by v4l2lb is used to dynamically add and remove v4l2lb video devices. It It uses new IOCTLs to do that, which I wonder if there is a better way.
* vim2m exposes config via a mechanism that doesn't require a new tool. v4l2lb requires a new control binary because of the custom ioctls.So, I have a couple questions:
* Would it be better to use vim2m as a model to implement something like v4l2loopback?
* If I were to do that should I copy vim2m to a new module, or should vim2m be extended. FWIW, vim2m appears to have two things against it:
* vim2m is a test v4l2 driver. It would need to be changed to build as a non-test driver. This is a pretty easyKconfigchange and maybe moving files to a more appropriate place in the linux source tree.
* v4l2loopback does have a number of features that vim2m does not have. I will say that I think that adding most importantant functionality (new formats and dynamic creation of loopback video devices in my mind) is probably very approachable and may give us a more modern linux module code.Give the above, I am kinda leaning toward extending vim2m code. So, do you have an opinion on the following? Should it be a new module, or should it literally extend and improve vim2m?
wt
Quoted 88 lines · Ezequiel Garcia
Hi Wren,
How about you submit the loopback driver for integration in mainline?
We can review the driver, give feedback and help you get it merged.
Ezequiel
On Mon, Mar 1, 2021, 5:32 PM Wren Turkal <[email protected] <mailto:[email protected]>> wrote:
Okay, so I have found how to enable the option to be manually selected.
Would a patch like the following be acceptable? Or is there a better way to do this?diff --git a/drivers/media/v4l2-core/Kconfig
b/drivers/media/v4l2-core/Kconfig
indexbf49f83cb86f..25004137a982100644
--- a/drivers/media/v4l2-core/Kconfig
+++ b/drivers/media/v4l2-core/Kconfig
@@ -54,7 +54,7 @@ configV4L2_H264# Used by drivers that need v4l2-mem2mem.ko config
V4L2_MEM2MEM_DEV
- tristate
+ tristate "V4L2 mem2mem framework support" depends onVIDEOBUF2_CORE# Used by LED subsystem flash drivers
Thanks,
wtOn 2/28/21 2:07 PM, Wren Turkal wrote:
> Hi there linux-media experts,
>
> I am working on the out-of-tree v4l2loopback module[1]. My > current goal is to implement the videobuf2 api. However, > it appears that I must use a differentvbuf2_queuefor the > OUTPUT and CAPTURE types. It appears that the mem2mem > framework has the ability to do just this. I came to this > conclusion while looking at the vim2m test module's code. > While trying to modify the loopback module, I have added > some code to try to callv4l2_m2m_init. You can see my very > WIP attempt at code here[2].
>
> I have installed the kernel-devel package on Fedora, which. > Unfortunately, I get he following error messages when trying > to compile:
>
> <errors>
> ERROR: modpost: "v4l2_m2m_release"
> [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined! > ERROR: modpost: "v4l2_m2m_init"
> [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined! > </errors>
>
> So, I looked into it, and Module.symvers and/proc/kallsyms> don't have those symbols listed, so I think the Fedora kernel > was not compiled with those symbols exported.
>
> Okay, so next I tried to build a custom kernel that does > export those symbols. I managed to get it done by enabling > the vim2m (VIDEO_VIM2Mconfig option) module. The new > Module.symvers includes the symbols; and the compile succeeds, > which brings me to my final act.
>
> I need to expose those symbols without enabling the vim2m > module since it is a testing module. I do not see a way > to do that explicitly. The config option in question
> appears to beV4L2_MEM2MEM_DEV. And I haven't found a > way to enable it without enabling something that implicitly > selects it (which vim2m does). Do y'all have any suggestions > for how to get these symbols exported properly when vim2m is > not enabled?
>
>
> Thanks,
> Wren Turkal (wt)
>
> [1] https://github.com/umlaeute/v4l2loopback
<https://github.com/umlaeute/v4l2loopback>
> [2] https://github.com/wt/v4l2loopback/tree/m2m_try
<https://github.com/wt/v4l2loopback/tree/m2m_try>
Questions below.
Quoted 8 lines · Nicolas Dufresne
No, since m2m are instance base, you cannot have two process operating on the same node (there is only 1 streaming state). You'd probably want to look at extending the VIMC driver instead. It's a virtual media controller, it's missing few bits to make loopback possible, but would be a good fit.
Note that vivid have a loopback capability, but it does not constraint the width/height/format and will produce a test pattern if you request something that differ from the source.
I am not sure which you're suggesting to use: vivid or vimc. Also, are you suggesting to extend them or use them as a template for a new module?
I think you are saying that vivid can already do loopback. However vivid is a test driver and, like vim2m, doesn't ship with distribution kernels. Should I try to move it out of the test-drivers?
Quoted 3 lines
It is very buggy, it often return a queued buffer back, and in fact, if you try to use the queue too much it fails. Though all this would probably be fine if you port it to VB2.
I am assuming you are saying to port v4l2loopback to vb2. I am working on that right now, but I am not sure how to proceed since the vb2 needs a queue for capture and a queue for output (since they cannot be the same). I have created the queues, but I am not sure how to further integrate. Please check out the WIP here:
https://github.com/wt/v4l2loopback/commit/e71b0a50427729a5e6e74443066751f7321dc404
If the m2m is the wrong approach, do you have any suggestions for how to proceed?
Also, I am currently making a change for the linux kernel moving this code info drivers/media/v4l2-core/v4l2-loopback.c so that I can start iterating with your feedback. Is that a correct place to put this code?
Thanks,
wt
Quoted 8 lines · Wren Turkal
Questions below.
Quoted 5 lines · Nicolas Dufresne
No, since m2m are instance base, you cannot have two process operating on the same node (there is only 1 streaming state). You'd probably want to look at extending the VIMC driver instead. It's a virtual media controller, it's missing few bits to make loopback possible, but would be a good fit.
Quoted 6 lines
Quoted 3 lines
Note that vivid have a loopback capability, but it does not constraint the width/height/format and will produce a test pattern if you request something that differ from the source.
I am not sure which you're suggesting to use: vivid or vimc. Also, are you suggesting to extend them or use them as a template for a new module?
I don't think either vimc nor vivid are particularly suitable for this, although vivid comes close. You can certainly lift ideas from vivid.
Quoted 15 lines
I think you are saying that vivid can already do loopback. However vivid is a test driver and, like vim2m, doesn't ship with distribution kernels. Should I try to move it out of the test-drivers?
Quoted 3 lines
It is very buggy, it often return a queued buffer back, and in fact, if you try to use the queue too much it fails. Though all this would probably be fine if you port it to VB2.
I am assuming you are saying to port v4l2loopback to vb2. I am working on that right now, but I am not sure how to proceed since the vb2 needs a queue for capture and a queue for output (since they cannot be the same). I have created the queues, but I am not sure how to further integrate. Please check out the WIP here:
https://github.com/wt/v4l2loopback/commit/e71b0a50427729a5e6e74443066751f7321dc404
Quoted 2 lines
If the m2m is the wrong approach, do you have any suggestions for how to proceed?
You need two video devices, one capture, one output. I see that the current loopback driver has just one video device and advertises it as a M2M device, but that's not what it should do. Creating two video devices (similar to what vivid does) is the best and most compatible approach.
Quoted 4 lines
Also, I am currently making a change for the linux kernel moving this code info
drivers/media/v4l2-core/v4l2-loopback.cso that I can start iterating with your feedback. Is that a correct place to put this code?
No, v4l2-core is for core V4L2 frameworks, not for drivers.
For now stick it in drivers/media/test-drivers. It will likely be moved again at some point (if we keep the 'misc' device that the current driver creates, then it might end up in drivers/media/misc).
By far the most important thing you need to do is convert it to vb2. Since the intention is to upstream the driver I would start by doing a cleanup of the code, ditching all compat code with older kernels, making sure it adheres to the linux coding style (use 'scripts/checkpatch.pl --strict' for that), then split it in two video devices and use vb2.
I actually wonder if it shouldn't be renamed to viloopback since it will be different from v4l2loopback. Anyway, that's something that can be done later.
Regards,
Hans
Quoted 4 lines
Thanks,
wt
Quoted 4 lines · Hans Verkuil
Quoted 2 lines
I am not sure which you're suggesting to use: vivid or vimc. Also, are you suggesting to extend them or use them as a template for a new module?
I don't think either vimc nor vivid are particularly suitable for this, although vivid comes close. You can certainly lift ideas from vivid.
Thanks for the advice. I'll definitely look at it more closely.
Quoted 7 lines
Quoted 7 lines
I am assuming you are saying to port v4l2loopback to vb2. I am working on that right now, but I am not sure how to proceed since the vb2 needs a queue for capture and a queue for output (since they cannot be the same). I have created the queues, but I am not sure how to further integrate. Please check out the WIP here:
https://github.com/wt/v4l2loopback/commit/e71b0a50427729a5e6e74443066751f7321dc404
Quoted 6 lines
Quoted 2 lines
If the m2m is the wrong approach, do you have any suggestions for how to proceed?
You need two video devices, one capture, one output. I see that the current loopback driver has just one video device and advertises it as a M2M device, but that's not what it should do. Creating two video devices (similar to what vivid does) is the best and most compatible approach.
FWIW, there is ongoing work to split the devices in v4l2loopback right now.
Quoted 9 lines
Quoted 3 lines
Also, I am currently making a change for the linux kernel moving this code info
drivers/media/v4l2-core/v4l2-loopback.cso that I can start iterating with your feedback. Is that a correct place to put this code?No, v4l2-core is for core V4L2 frameworks, not for drivers.
For now stick it in
drivers/media/test-drivers. It will likely be moved again at some point (if we keep the 'misc' device that the current driver creates, then it might end up indrivers/media/misc).
Quoted 8 lines
By far the most important thing you need to do is convert it to vb2. Since the intention is to upstream the driver I would start by doing a cleanup of the code, ditching all compat code with older kernels, making sure it adheres to the linux coding style (use '
scripts/checkpatch.pl--strict' for that), then split it in two video devices and use vb2.I actually wonder if it shouldn't be renamed to viloopback since it will be different from v4l2loopback. Anyway, that's something that can be done later.
Thanks for the advice. I'm happy to rename it to viloopback now if that makes sense. That is very little work.
I am moving the code to test-drivers for now. Thanks for that feedback.
wt