Skip to contentSkip to conversation

libv4l: Add plugin support

PATCH Superseded Yordan Kamenov 2 participants 4 messages Jan 7, 2011 – Jan 11, 2011 Media (V4L/DVB)

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

cover libv4l: Add plugin support

Hi Hans,

Here is initial version of plugin support for libv4l, based on your RFC.

It is provided by functions v4l2_plugin_[open,close,etc]. When open() is called libv4l dlopens files in /usr/lib/libv4l/plugins 1 at a time and call open() callback passing through the applications parameters unmodified. If a plugin is relevant for the specified device node, it can indicate so by returning a value other then -1 (the actual file descriptor).

As soon as a plugin returns another value then -1 plugin loading stops and information about it (fd and corresponding library handle) is stored. For each function v4l2_[ioctl,read,close,etc] is called corresponding v4l2_plugin_* function which looks if there is loaded plugin for that file and call it's callbacks. v4l2_plugin_* functions indicate by their first argument if plugin was used, and if it was not then v4l2_* functions proceed with their usual behavior.

Yordan Kamenov (1):
Add plugin support to libv4l

 lib/include/libv4l2-plugin.h |   74 ++++++++
 lib/include/libv4l2.h        |   15 ++
 lib/libv4l2/Makefile         |    4 +-
 lib/libv4l2/libv4l2-priv.h   |    9 +
 lib/libv4l2/libv4l2.c        |   56 ++++++-
 lib/libv4l2/v4l2-plugin.c    |  399 ++++++++++++++++++++++++++++++++++++++++++
 lib/libv4l2/v4l2convert.c    |   20 ++-
 7 files changed, 568 insertions(+), 9 deletions(-)
 create mode 100644 lib/include/libv4l2-plugin.h
 create mode 100644 lib/libv4l2/v4l2-plugin.c
Signature · 1 line
1.7.3.1
#1 lore raw

PATCH 1/1 Add plugin support to libv4l

A libv4l2 plugin will sit in between libv4l2 itself and the actual /dev/video device node a fd refers to. It will be called each time libv4l2 wants to do an operation (read/write/ioctl/mmap/munmap) on the actual /dev/video node in question.

---
 lib/include/libv4l2-plugin.h |   74 ++++++++
 lib/include/libv4l2.h        |   15 ++
 lib/libv4l2/Makefile         |    4 +-
 lib/libv4l2/libv4l2-priv.h   |    9 +
 lib/libv4l2/libv4l2.c        |   56 ++++++-
 lib/libv4l2/v4l2-plugin.c    |  399 ++++++++++++++++++++++++++++++++++++++++++
 lib/libv4l2/v4l2convert.c    |   20 ++-
 7 files changed, 568 insertions(+), 9 deletions(-)
 create mode 100644 lib/include/libv4l2-plugin.h
 create mode 100644 lib/libv4l2/v4l2-plugin.c
lib/include/libv4l2-plugin.h new file +74 -0
new file mode 100644
index 0000000..881b55d
@@ -0,0 +1,74 @@
+/*
+* Copyright (C) 2010 Nokia Corporation <[email protected]>
+
+* This program is free software; you can redistribute it and/or modify
+* it under the terms of the GNU Lesser General Public License as published by
+* the Free Software Foundation; either version 2.1 of the License, or
+* (at your option) any later version.
+*
+* This program is distributed in the hope that it will be useful,
+* but WITHOUT ANY WARRANTY; without even the implied warranty of
+* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+* Lesser General Public License for more details.
+*
+* You should have received a copy of the GNU Lesser General Public License
+* along with this program; if not, write to the Free Software
+* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+*/
+
+#ifndef __LIBV4L2_PLUGIN_H
+#define __LIBV4L2_PLUGIN_H
+
+#include <sys/types.h>
+
+/* A libv4l2 plugin will sit in between libv4l2 itself and the
+   actual /dev/video device node a fd refers to. It will be called each time
+   libv4l2 wants to do an operation (read/write/ioctl/mmap/munmap) on the
+   actual /dev/video node in question. When called the plugin can then choose
+   to do one of the following:
+   1. Pass the call unmodified to the fd, and return the return value unmodifed
+   2. Modify some arguments in the call and pass it through
+   3. Modify the return(ed) value(s) of a passed through call
+   4. Not do any operation on the fd at all but instead completely fake it
+       (which opens the possibility for "fake" v4l devices)
+
+   libv4l2 plugins should *never* use any global variables. All data should be
+   bound to the specific fd to which the plugin is bound. This ensures that for
+   example a plugin for a specific type of usb webcam will also work when 2
+   identical cameras are plugged into a system (and both are used from the same
+   process).
+
+   A libv4l2 plugin can register plugin private data using:
+   void libv4l2_set_plugindata(int fd, void *plugin_data);
+
+   And can get this data out of libv4l2 again inside a callback using:
+   void *libv4l2_get_plugindata(int fd);
+
+   Note that a plugin should call libv4l2_set_plugindata only once per fd !
+   Calling it a second time will overwrite the previous value. The logical
+   place to use libv4l2_set_plugindata is from the plugin's open callback.
+*/
+
+/* Plugin callback function struct */
+struct libv4l2_plugin_data {
+	int (*open)(const char *file, int oflag, ...);
+	int (*close)(int fd);
+	int (*ioctl)(int fd, unsigned long int request, ...);
+	ssize_t (*read)(int fd, void *buffer, size_t n);
+	void *(*mmap)(void *start, size_t length, int prot, int flags,
+			int fd, int64_t offset);
+	/* Note as munmap has no fd argument, defining a callback for munmap
+		will result in it getting called for *any* call to v4l2_munmap.
+		So if a plugin defines a callback for munmap (because for
+		example it returns fake mmap buffers from its mmap callback).
+		Then it must keep track of the addresses at which these buffers
+		live and their size and check the munmap arguments to see if the
+		munmap call was meant for it. */
+	int (*munmap)(void *_start, size_t length);
+};
+
+/* Plugin utility functions */
+void libv4l2_set_plugindata(int fd, void *plugin_data);
+void *libv4l2_get_plugindata(int fd);
+
+#endif
lib/include/libv4l2.h +15 -0
index cc0ab4a..2123546 100644
@@ -22,6 +22,7 @@
 #include <stdio.h>
 #include <unistd.h>
 #include <stdint.h>
+#include "libv4l2-plugin.h"
 
 #ifdef __cplusplus
 extern "C" {
@@ -108,6 +109,20 @@ LIBV4L_PUBLIC int v4l2_get_control(int fd, int cid);
    (note the fd is left open in this case). */
 LIBV4L_PUBLIC int v4l2_fd_open(int fd, int v4l2_flags);
 
+
+LIBV4L_PUBLIC int v4l2_plugin_open(int *plugin_used, const char *file,
+					int oflag, ...);
+LIBV4L_PUBLIC int v4l2_plugin_close(int *plugin_used, int fd);
+LIBV4L_PUBLIC int v4l2_plugin_ioctl(int *plugin_used, int fd,
+					unsigned long int request, ...);
+LIBV4L_PUBLIC ssize_t v4l2_plugin_read(int *plugin_used, int fd, void *dest,
+					size_t n);
+LIBV4L_PUBLIC void *v4l2_plugin_mmap(int *plugin_used, void *start,
+					size_t length, int prot, int flags,
+					int fd, int64_t offset);
+LIBV4L_PUBLIC int v4l2_plugin_munmap(int *plugin_used, void *_start,
+					size_t length);
+
 #ifdef __cplusplus
 }
 #endif /* __cplusplus */
lib/libv4l2/Makefile +2 -2
index d78632f..eb1c019 100644
@@ -1,8 +1,8 @@
 override CPPFLAGS += -I../include -fvisibility=hidden
 
-LIBS_libv4l2  = -lpthread
+LIBS_libv4l2  = -lpthread -ldl
 
-V4L2_OBJS     = libv4l2.o log.o
+V4L2_OBJS     = libv4l2.o v4l2-plugin.o log.o
 V4L2CONVERT   = v4l2convert.so
 V4L2CONVERT_O = v4l2convert.o libv4l2.so
 TARGETS       = $(V4L2_LIB) libv4l2.pc
lib/libv4l2/libv4l2-priv.h +9 -0
index 46d6103..3873b1d 100644
@@ -87,6 +87,15 @@ struct v4l2_dev_info {
 	unsigned char *readbuf;
 };
 
+struct v4l2_plugin_info {
+	int fd;
+	void *plugin_library;
+	struct libv4l2_plugin_data *libv4l2_plugin;
+	unsigned char *frame_pointers[V4L2_MAX_NO_FRAMES];
+	int frame_sizes[V4L2_MAX_NO_FRAMES];
+	void *plugin_data;
+};
+
 /* From log.c */
 void v4l2_log_ioctl(unsigned long int request, void *arg, int result);
 
lib/libv4l2/libv4l2.c +50 -6

112 lines not on this page · raw message

lib/libv4l2/v4l2-plugin.c new file +399 -0

399 lines not on this page · raw message

lib/libv4l2/v4l2convert.c +19 -1
index e251085..9f28490 100644
@@ -31,6 +31,7 @@
 #include "../libv4lconvert/libv4lsyscall-priv.h"
 #include <linux/videodev2.h>
 #include <libv4l2.h>
+#include "libv4l2-plugin.h"
 
 /* Check that open/read/mmap is not a define */
 #if defined open || defined read || defined mmap
@@ -45,10 +46,27 @@
 
 LIBV4L_PUBLIC int open(const char *file, int oflag, ...)
 {
-	int fd;
+	int fd, plugin_used;
 	struct v4l2_capability cap;
 	int v4l_device = 0;
 
+	if (oflag & O_CREAT) {
+		va_list ap;
+		mode_t mode;
+
+		va_start(ap, oflag);
+		mode = va_arg(ap, mode_t);
+
+		fd = v4l2_plugin_open(&plugin_used, file, oflag, mode);
+
+		va_end(ap);
+	} else {
+		fd = v4l2_plugin_open(&plugin_used, file, oflag, 0);
+	}
+
+	if (plugin_used)
+		return fd;
+
 	/* check if we're opening a video4linux2 device */
 	if (!strncmp(file, "/dev/video", 10) || !strncmp(file, "/dev/v4l/", 9)) {
 		/* Some apps open the device read only, but we need rw rights as the
Signature · 1 line
1.7.3.1
#2 lore raw

Re: [PATCH 1/1] Add plugin support to libv4l

Hi,

First of all many thanks for working on this! I've several remarks which I would like to see addressed before merging this.

Since most remarks are rather high level remarks I've opted to just make a bulleted list of them rather then inserting them inline.

* The biggest problem with your current implementation is that for each existing libv4l2_foo function you check if there is a plugin attached to the fd passed in and if that plugin wants to handle the call. Now lets assume that there is a plugin and that it wants to handle all calls. That means that you've now effectively replaced all libv4l2_foo calls with calling the corresponding foo function from the plugin and returning its result. This means that for this fd / device you've achieved the same result as completely replacing libv4l2.so.0 with a new library containing the plugin code.

IOW you've not placed then plugin between libv4l2 and the device (as intended) but completely short-circuited / replaced libv4l2. This means for example for a device which only supports yuv output, that libv4l2 will no longer do format emulation and conversion and an app which only supports devices which deliver rgb data will no longer work.

To actually place the plugin between libv4l2 (and libv4lconvert) and the device, you should replace all the SYS_FOO calls in libv4l2. The SYS_FOO calls are the calls to the actual device, so be replacing those with calls to the plugin you actual place the plugin between libv4l and the device as intended.

* Currently you add a loop much like the one in the v4l2_get_index function to each libv4l2_plugin function. Basically you add an array of v4l2_plugin_info structs in libv4l2-plugin. Which gets searched by fd, much like the v4l2_dev_info struct array. Including needing similar locking. I would like you to instead just store the plugin info for a certain fd directly into the v4l2_dev_info struct. This way the separate array, looping and locking can go away.

* Next I would also like to see all the libv4l2_plugin_foo functions except for libv4l2_plugin_open go away. Instead libv4l2.c can call the plugin functions directly. Let me try to explain what I have in mind. Lets say we store the struct libv4l2_plugin_data pointer to the active plugin in the v4l2_dev_info struct and name it dev_ops (short for device operations).

Then we can replace all SYS_FOO calls inside libv4l2 (except the ones were v4l2_get_index returns -1), with a call to the relevant devop functions, for example:

                result = SYS_IOCTL(devices[index].fd, VIDIOC_REQBUFS, req);
Would become:
                result = devices[index].dev_ops->v4l2_plugin_ioctl(
                                    devices[index].fd, VIDIOC_REQBUFS, req);

Note that the plugin_used parameter of the v4l2_plugin_ioctl is gone, it should simply do a normal SYS_IOCTL and return the return value of that if it is not interested in intercepting the ioctl (you could move the definition of the SYS_FOO macros to libv4l2-plugin.h to make them availables to plugins).

Also I think it would be better to rename the function pointers inside the libv4l2_plugin_data struct from v4l2_plugin_foo to just foo, so that the above code would become:

                result = devices[index].dev_ops->v4l2_plugin_ioctl(
                                    devices[index].fd, VIDIOC_REQBUFS, req);

* The above means that need to always have a dev_ops pointer, so we need to have a default_dev_ops struct to use when no plugin wants to talk to the device.

* You've put the v4l2_plugin_foo functions (of which only v4l2_plugin_foo will remain in my vision) in lib/include/libv4l2.h I don't think these functions should be public, their prototypes should be moved to lib/libv4l2/libv4l2-priv.h, and they should not be declared LIBV4L_PUBLIC.

* There is one special case in all this, files under libv4lconvert also use SYS_IOCTL in various places. Since this now need to go through the plugin we need to take some special measures here. There are 2 options:
1) Break the libv4lconvert ABI (very few programs use it) and pass a struct libv4l2_plugin_data pointer to the v4lconvert_create function. *And* export the default_dev_ops struct from libv4l2.
2) Add a libv4l2_raw_ioctl method, which just gets the index and then does devices[index].dev_ops->v4l2_plugin_ioctl
Except that this is not really an option as libv4lconvert should not depend on libv4l2
My vote personally goes to 1.

* I think that once we do 1) from above it would be good to rename libv4l2_plugin_data to libv4l2_dev_ops, as that makes the public API more clear and dev_ops is in essence what a plugin provides.

* Note that were I wrote: "like to see all the libv4l2_plugin_foo functions except for libv4l2_plugin_open go away" I did so for simplicity, in reality the wrappers around mmap and munmap need to stay too, but they should use data directly stored inside the v4l2_dev_info struct. This means that we need to either:
mv the mmap and munmap code to libv4l2.c; or export the v4l2_dev_info struct array. I vote for exporting the v4l2_dev_info struct array (through libv4l2-priv.h, so it won't be visible to the outside world, but it will be usable outside libv4l2.c).

Thanks & Regards,

Hans

#3 lore raw

Re: [PATCH 1/1] Add plugin support to libv4l

Hi Hans,

Thanks for your comments.

Quoted 24 lines · Hans de Goede

Hi,

First of all many thanks for working on this! I've several remarks which I would like to see addressed before merging this.

Since most remarks are rather high level remarks I've opted to just make a bulleted list of them rather then inserting them inline.

* The biggest problem with your current implementation is that for each existing libv4l2_foo function you check if there is a plugin attached to the fd passed in and if that plugin wants to handle the call. Now lets assume that there is a plugin and that it wants to handle all calls. That means that you've now effectively replaced all libv4l2_foo calls with calling the corresponding foo function from the plugin and returning its result. This means that for this fd / device you've achieved the same result as completely replacing libv4l2.so.0 with a new library containing the plugin code.

IOW you've not placed then plugin between libv4l2 and the device (as intended) but completely short-circuited / replaced libv4l2. This means for example for a device which only supports yuv output, that libv4l2 will no longer do format emulation and conversion and an app which only supports devices which deliver rgb data will no longer work.

Quoted 5 lines

To actually place the plugin between libv4l2 (and libv4lconvert) and the device, you should replace all the SYS_FOO calls in libv4l2. The SYS_FOO calls are the calls to the actual device, so be replacing those with calls to the plugin you actual place the plugin between libv4l and the device as intended.

I agree with that, currently the plugin can replace the libv4l2, but if we replace SYS_FOO calls it will actually sit between library and the video node. I will do that.

Quoted 8 lines

* Currently you add a loop much like the one in the v4l2_get_index function to each libv4l2_plugin function. Basically you add an array of v4l2_plugin_info structs in libv4l2-plugin. Which gets searched by fd, much like the v4l2_dev_info struct array. Including needing similar locking. I would like you to instead just store the plugin info for a certain fd directly into the v4l2_dev_info struct. This way the separate array, looping and locking can go away.

I have put separate array, because the array of v4l2_dev_info is declared static in libv4l2.c and is not visible to v4l2-plugin.c (I did't want to change it's declaration). But with changes that you suggest below, there is no problem to add plugin data to v4l2_dev_info.

Quoted 27 lines

* Next I would also like to see all the libv4l2_plugin_foo functions except for libv4l2_plugin_open go away. Instead libv4l2.c can call the plugin functions directly. Let me try to explain what I have in mind. Lets say we store the struct libv4l2_plugin_data pointer to the active plugin in the v4l2_dev_info struct and name it dev_ops (short for device operations).

Then we can replace all SYS_FOO calls inside libv4l2 (except the ones were v4l2_get_index returns -1), with a call to the relevant devop functions, for example:
result = SYS_IOCTL(devices[index].fd, VIDIOC_REQBUFS,
req);
Would become:

                result = devices[index].dev_ops->v4l2_plugin_ioctl(
                                    devices[index].fd, VIDIOC_REQBUFS, 

req);

Note that the plugin_used parameter of the v4l2_plugin_ioctl is gone, it should simply do a normal SYS_IOCTL and return the return value of that if it is not interested in intercepting the ioctl (you could move the definition of the SYS_FOO macros to libv4l2-plugin.h to make them availables to plugins).

Also I think it would be better to rename the function pointers inside the libv4l2_plugin_data struct from v4l2_plugin_foo to just foo, so that the above code would become:

Quoted 7 lines
                result = devices[index].dev_ops->v4l2_plugin_ioctl(
                                    devices[index].fd, VIDIOC_REQBUFS, 

req);

* The above means that need to always have a dev_ops pointer, so we need to have a default_dev_ops struct to use when no plugin wants to talk to the device.

Ok, I will replace SYS_FOO calls with dev_ops structure.

Quoted 23 lines

* You've put the v4l2_plugin_foo functions (of which only v4l2_plugin_foo will remain in my vision) in lib/include/libv4l2.h I don't think these functions should be public, their prototypes should be moved to lib/libv4l2/libv4l2-priv.h, and they should not be declared LIBV4L_PUBLIC.

* There is one special case in all this, files under libv4lconvert also use SYS_IOCTL in various places. Since this now need to go through the plugin we need to take some special measures here. There are 2 options:
1) Break the libv4lconvert ABI (very few programs use it) and pass a struct libv4l2_plugin_data pointer to the v4lconvert_create function. *And* export the default_dev_ops struct from libv4l2.
2) Add a libv4l2_raw_ioctl method, which just gets the index and then does devices[index].dev_ops->v4l2_plugin_ioctl
Except that this is not really an option as libv4lconvert should not depend on libv4l2
My vote personally goes to 1.

* I think that once we do 1) from above it would be good to rename libv4l2_plugin_data to libv4l2_dev_ops, as that makes the public API more clear and dev_ops is in essence what a plugin provides.

Quoted 9 lines

* Note that were I wrote: "like to see all the libv4l2_plugin_foo functions except for libv4l2_plugin_open go away" I did so for simplicity, in reality the wrappers around mmap and munmap need to stay too, but they should use data directly stored inside the v4l2_dev_info struct. This means that we need to either:
mv the mmap and munmap code to libv4l2.c; or export the v4l2_dev_info struct array. I vote for exporting the v4l2_dev_info struct array (through libv4l2-priv.h, so it won't be visible to the outside world, but it will be usable outside libv4l2.c).

Ok, I will make 1) and export v4l2_dev_info.

Quoted 5 lines

Thanks & Regards,

Hans

Best Regards
Yordan