Mem-to-mem device framework
Subsystems (2)
cover Mem-to-mem device framework
Hello,
this is the second version of the proposed implementation for mem-to-mem memory device framework. Your comments are very welcome.
In v2.1:
I am very sorry for the resend, but somehow an orphaned endif found its way to Kconfig during the rebase.
Changes since v1:
- v4l2_m2m_buf_queue() now requires m2m_ctx as its argument
- video_queue private data stores driver private data
- a new submenu in kconfig for mem-to-mem devices
- minor rebase leftovers cleanup
A second patch series followed v2 with a new driver for a real device - Samsung S3C/S5P image rotator, utilizing this framework.
This series contains:
[PATCH v2.1 1/2] V4L: Add memory-to-memory device helper framework for V4L2. [PATCH v2.1 2/2] V4L: Add a mem-to-mem V4L2 framework test device. [EXAMPLE v2] Mem-to-mem userspace test application.
Previous discussion and RFC on this topic:
http://thread.gmane.org/gmane.linux.drivers.video-input-infrastructure/10668
A mem-to-mem device is a device that uses memory buffers passed by userspace applications for both source and destination. This is different from existing drivers that use memory buffers for only one of those at once.
In terms of V4L2 such a device would be both of OUTPUT and CAPTURE type. Although no such devices are present in the V4L2 framework, a demand for such a model exists, e.g. for 'resizer devices'.
-------------------------------------------------------------------------------
Mem-to-mem devices
-------------------------------------------------------------------------------
In the previous discussion we concluded that we should use one video node with two queues, an output (V4L2_BUF_TYPE_VIDEO_OUTPUT) queue for source buffers and a capture queue (V4L2_BUF_TYPE_VIDEO_CAPTURE) for destination buffers.
Each instance has its own set of queues: 2 videobuf_queues, each with a ready buffer queue, managed by the framework. Everything is encapsulated in the queue context struct:
struct v4l2_m2m_queue_ctx {
struct videobuf_queue q;
/* ... */
/* Queue for buffers ready to be processed as soon as this
* instance receives access to the device */
struct list_head rdy_queue;
/* ... */
};struct v4l2_m2m_ctx {
/* ... */
/* Capture (output to memory) queue context */
struct v4l2_m2m_queue_ctx cap_q_ctx;
/* Output (input from memory) queue context */
struct v4l2_m2m_queue_ctx out_q_ctx;
/* ... */
};Streamon can be called for all instances and will not sleep if another instance is streaming.
vidioc_querycap() should report V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_VIDEO_OUTPUT.
-------------------------------------------------------------------------------
Queuing and dequeuing buffers
-------------------------------------------------------------------------------
Applications can queue as many buffers as they want and it is not required to queue an equal number of source and destination buffers. If there is not enough buffers of any type, a new transaction will simply not be scheduled.
-------------------------------------------------------------------------------
Source and destination formats
-------------------------------------------------------------------------------
Should be set per queue. A helper function to access queues depending on the passed type - v4l2_m2m_get_vq() - is supplied. Most of the format-handling code is normally located in drivers anyway. The only exception is the "field" member of the videobuf_queue struct, which has to be set directly. It breaks encapsulation a little bit, but nothing can be done with it.
-------------------------------------------------------------------------------
Scheduling
-------------------------------------------------------------------------------
Requirements/assumptions:
1. More than one instance can be open at the same time.
2. Each instance periodically receives exclusive access to the device, performs an operation (operations) and yields back the device in a state that allows other instances to use it.
3. When an instance gets access to the device, it performs a "transaction"/"job". A transaction/job is defined as the shortest operation that cannot/should not be further divided without having to restart it from scratch, or without having to perform expensive reconfiguration of a device, etc.
4. Transactions can use multiple source/destination buffers.
5. Only a driver can tell when it is ready to perform a transaction, so a optional callback is provided for that purpose (job_ready()).
There are three common requirements for a transaction to be ready to run:
- at least one source buffer ready
- at least one destination buffer ready
- streaming on
- (optional) driver-specific requirements (driver-specific callback function)
So when buffers are queued by qbuf() or streaming is turned on with streamon(), the framework calls v4l2_m2m_try_schedule().
v4l2_m2m_try_schedule()
1. Checks for the above conditions.
2. Checks for driver-specific conditions by calling job_ready() callback, if supplied.
3. If all the checks succeed, it calls v4l2_m2m_schedule() to schedule the transaction.
v4l2_m2m_schedule()
1. Checks whether the transaction is already on job queue and schedules it if not (by adding it to the job queue).
2. Calls v4l2_m2m_try_run().
v4l2_m2m_try_run()
1. Runs a job if and is pending and none is currently running by calling device_run() callback.
When the device_run() callback is called, the driver has to begin the transaction. When it is finished, the driver has to call v4l2_m2m_job_finish().
v4l2_m2m_job_finish()
1. Removes the currently running transaction from the job queue and calls v4l2_m2m_try_run to (possibly) run the next pending transaction.
There is also support for forced transaction aborting (when an application gets killed). The framework calls job_abort() callback and the driver has to abort the transaction as soon as possible and call v4l2_m2m_job_finish() to indicate that the transaction has been aborted.
Additionally, some kind of timeout for transactions could be added to prevent instances from claiming the device for too long.
-------------------------------------------------------------------------------
Acquiring ready buffers to process
-------------------------------------------------------------------------------
Ready buffers can be acquired using v4l2_m2m_next_src_buf()/ v4l2_m2m_next_dst_buf(). After the transaction they are removed from the queues with v4l2_m2m_dst_buf_remove()/v4l2_m2m_src_buf_remove(). This is not multi-buffer-transaction-safe. It will have to be modified, but ideally after we decide how to handle multi-buffer transactions in videobuf core.
-------------------------------------------------------------------------------poll()
-------------------------------------------------------------------------------
We cannot have poll() for multiple queues on one node, so we use poll() for the destination queue only.
-------------------------------------------------------------------------------mmap()
-------------------------------------------------------------------------------
Requirements:
- allow mapping buffers from different queues
- retain "magic" offset values so videobuf can still match buffers by offsets
The proposed solution involves a querybuf() and mmap() multiplexers:
a) When a driver calls querybuf(), we have access to the type and we can detect which queue to call videobuf_querybuf() on:
vq = v4l2_m2m_get_vq(m2m_ctx, buf->type);
ret = videobuf_querybuf(vq, buf);The offsets returned from videobuf_querybuf() for one of the queues are further offset by a predefined constant (DST_QUEUE_OFF_BASE). This way the driver (and applications) receive different offsets for the same buffer indexes of each queue:
if (buf->memory == V4L2_MEMORY_MMAP
&& vq->type == V4L2_BUF_TYPE_VIDEO_CAPTURE) {
buf->m.offset += DST_QUEUE_OFF_BASE;
}b) When the application (driver) calls mmap(), the offsets which were modified in querybuf() are detected and the proper queue for them chosen based on that. Finally, the modified offsets are passed to videobuf_mmap_mapper() for proper queues with their offsets changed back to values recognizable by videobuf:
unsigned long offset = vma->vm_pgoff << PAGE_SHIFT;
struct videobuf_queue *vq;
if (offset < DST_QUEUE_OFF_BASE) {
vq = v4l2_m2m_get_src_vq(m2m_ctx);
} else {
vq = v4l2_m2m_get_dst_vq(m2m_ctx);
vma->vm_pgoff -= (DST_QUEUE_OFF_BASE >> PAGE_SHIFT);
}
return videobuf_mmap_mapper(vq, vma);-------------------------------------------------------------------------------
Test device and a userspace application
-------------------------------------------------------------------------------mem2mem_testdev.c is a test driver for the framework. It uses timers for fake interrupts and allows testing transaction with different number of buffers and transaction durations simultaneously.
process-vmalloc.c is a capture+output test application for the test device.
-------------------------------------------------------------------------------
Future work
-------------------------------------------------------------------------------
- read/write support
- transaction/abort timeouts
- extracting more common code to the framework? (e.g. per-queue format details, transaction length, etc.)
Best regards
--
Pawel Osciak
Linux Platform Group
Samsung Poland R&D Center
PATCH 1/2 V4L: Add memory-to-memory device helper framework for V4L2.
A mem-to-mem device is a device that uses memory buffers passed by userspace applications for both source and destination data. This is different from existing drivers, which use memory buffers for only one of those at once.
In terms of V4L2 such a device would be both of OUTPUT and CAPTURE type. Although no such devices are present in the V4L2 framework, a demand for such a model exists, e.g. for 'resizer devices'.
This patch also adds a separate kconfig submenu for mem-to-mem V4L devices.
--- drivers/media/video/Kconfig | 14 + drivers/media/video/Makefile | 2 + drivers/media/video/v4l2-mem2mem.c | 671 ++++++++++++++++++++++++++++++++++++ include/media/v4l2-mem2mem.h | 153 ++++++++ 4 files changed, 840 insertions(+), 0 deletions(-) create mode 100644 drivers/media/video/v4l2-mem2mem.c create mode 100644 include/media/v4l2-mem2mem.h
drivers/media/video/Kconfig +14 -0
@@ -45,6 +45,10 @@ config VIDEO_TUNER tristate depends on MEDIA_TUNER +config V4L2_MEM2MEM_DEV + tristate + depends on VIDEOBUF_GEN + # # Multimedia Video device configuration #
@@ -1075,3 +1079,13 @@ config USB_S2255 endif # V4L_USB_DRIVERS endif # VIDEO_CAPTURE_DRIVERS + +menuconfig V4L_MEM2MEM_DRIVERS + bool "Memory-to-memory multimedia devices" + depends on VIDEO_V4L2 + default n + ---help--- + Say Y here to enable selecting drivers for V4L devices that + use system memory for both source and destination buffers, as opposed + to capture and output drivers, which use memory buffers for just + one of those.
drivers/media/video/Makefile +2 -0
@@ -115,6 +115,8 @@ obj-$(CONFIG_VIDEOBUF_VMALLOC) += videobuf-vmalloc.o obj-$(CONFIG_VIDEOBUF_DVB) += videobuf-dvb.o obj-$(CONFIG_VIDEO_BTCX) += btcx-risc.o +obj-$(CONFIG_V4L2_MEM2MEM_DEV) += v4l2-mem2mem.o + obj-$(CONFIG_VIDEO_M32R_AR_M64278) += arv.o obj-$(CONFIG_VIDEO_CX2341X) += cx2341x.o
drivers/media/video/v4l2-mem2mem.c new file +671 -0
671 lines not on this page · raw message
include/media/v4l2-mem2mem.h new file +153 -0
153 lines not on this page · raw message
Signature · 1 line
1.6.4.2.253.g0b1fac
PATCH 1/2 V4L: Add memory-to-memory device helper framework for V4L2.
Quoted 9 lines · Pawel Osciak
A mem-to-mem device is a device that uses memory buffers passed by userspace applications for both source and destination data. This is different from existing drivers, which use memory buffers for only one of those at once.
In terms of V4L2 such a device would be both of OUTPUT and CAPTURE type. Although no such devices are present in the V4L2 framework, a demand for such a model exists, e.g. for 'resizer devices'.
Quoted 5 lines
This patch also adds a separate kconfig submenu for mem-to-mem V4L devices.
Signed-off-by Pawel Osciak <[email protected]>Signed-off-by Marek Szyprowski <[email protected]>Reviewed-by Kyungmin Park <[email protected]>
Pawel,
I did find a few things that I want to mention. (If you think I'm wrong on something feel free to say so, I was interrupted several times when lpoking things over.)
Quoted 116 lines
--- drivers/media/video/Kconfig | 14 + drivers/media/video/Makefile | 2 + drivers/media/video/v4l2-mem2mem.c | 671 ++++++++++++++++++++++++++++++++++++ include/media/v4l2-mem2mem.h | 153 ++++++++ 4 files changed, 840 insertions(+), 0 deletions(-) create mode 100644 drivers/media/video/v4l2-mem2mem.c create mode 100644 include/media/v4l2-mem2mem.hdrivers/media/video/Kconfig +14
-0@@ -45,6 +45,10 @@ config VIDEO_TUNER tristate depends on MEDIA_TUNER +config V4L2_MEM2MEM_DEV + tristate + depends on VIDEOBUF_GEN + # # Multimedia Video device configuration #@@ -1075,3 +1079,13 @@ config USB_S2255 endif # V4L_USB_DRIVERS endif # VIDEO_CAPTURE_DRIVERS + +menuconfig V4L_MEM2MEM_DRIVERS + bool "Memory-to-memory multimedia devices" + depends on VIDEO_V4L2 + default n + ---help--- + Say Y here to enable selecting drivers for V4L devices that + use system memory for both source and destination buffers, as opposed + to capture and output drivers, which use memory buffers for just + one of those.drivers/media/video/Makefile +2
-0@@ -115,6 +115,8 @@ obj-$(CONFIG_VIDEOBUF_VMALLOC) += videobuf-vmalloc.o obj-$(CONFIG_VIDEOBUF_DVB) += videobuf-dvb.o obj-$(CONFIG_VIDEO_BTCX) += btcx-risc.o +obj-$(CONFIG_V4L2_MEM2MEM_DEV) += v4l2-mem2mem.o + obj-$(CONFIG_VIDEO_M32R_AR_M64278) += arv.o obj-$(CONFIG_VIDEO_CX2341X) += cx2341x.odrivers/media/video/v4l2-mem2mem.c new file +59
-0@@ -0,0 +1,671 @@ +/* +* Memory-to-memory device framework for Video for Linux 2. +* +* Helper functions for devices that use memory buffers for both source +* and destination. +* +* Copyright (c) 2009 Samsung Electronics Co., Ltd. +* Pawel Osciak, <[email protected]> +* Marek Szyprowski, <[email protected]> +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by the +* Free Software Foundation; either version 2 of the +* License, or (at your option) any later version +*/ + +#include <linux/module.h> +#include <linux/sched.h> +#include <media/videobuf-core.h> +#include <media/v4l2-mem2mem.h> + +MODULE_DESCRIPTION ("Mem to mem device framework for V4L2" ); +MODULE_AUTHOR ("Pawel Osciak, <[email protected]>" ); +MODULE_LICENSE ("GPL" ); + +static int debug; +module_param (debug, int,0644 ); + +#define dprintk(fmt, arg...) do {\ + if (debug >= 1)\ + printk(KERN_DEBUG "%s: " fmt, __func__, ## arg); } while (0) + + +/* The instance is already queued on the jobqueue */ +#define TRANS_QUEUED (1 << 0) +/* The instance is currently running in hardware */ +#define TRANS_RUNNING (1 << 1) + + +/* Offset base for buffers on the destination queue - used to distinguish +* between source and destination buffers when mmapping - they receive the same +* offsets but for different queues */ +#define DST_QUEUE_OFF_BASE (TASK_SIZE / 2) + + +struct v4l2_m2m_dev { +/* Currently running instance */ +struct v4l2_m2m_ctx *curr_ctx; +/* Instances queued to run */ +struct list_head jobqueue; +spinlock_t job_spinlock; + +struct v4l2_m2m_ops *m2m_ops; +}; + +static inline +struct v4l2_m2m_queue_ctx *get_queue_ctx (struct v4l2_m2m_ctx *m2m_ctx, +enum v4l2_buf_type type) +{
drivers/media/video/v4l2-mem2mem.c line 69
Quoted 10 lines
drivers/media/video/v4l2-mem2mem.c new file +10
-0+switch (type) { +case V4L2_BUF_TYPE_VIDEO_CAPTURE : +return &m2m_ctx->cap_q_ctx; +case V4L2_BUF_TYPE_VIDEO_OUTPUT : +return &m2m_ctx->out_q_ctx; +default : +printk (KERN_ERR "Invalid buffer type\n" ); +return NULL ; + } +}
The logic above is fine. I'm just surprised gcc doesn't gripe about "control reaching the end of a non-void function". I guess gcc has gotten smarter.
Quoted 40 lines
drivers/media/video/v4l2-mem2mem.c new file +40
-0+/** +* v4l2_m2m_get_vq() - return videobuf_queue for the given type +*/ +struct videobuf_queue *v4l2_m2m_get_vq (struct v4l2_m2m_ctx *m2m_ctx, +enum v4l2_buf_type type) +{ +struct v4l2_m2m_queue_ctx *q_ctx; + + q_ctx =get_queue_ctx (m2m_ctx, type); +if (!q_ctx) +return NULL ; + +return &q_ctx->q; +} +EXPORT_SYMBOL (v4l2_m2m_get_vq); + +/** +* v4l2_m2m_get_src_vq() - return videobuf_queue for source buffers +*/ +struct videobuf_queue *v4l2_m2m_get_src_vq (struct v4l2_m2m_ctx *m2m_ctx) +{ +return v4l2_m2m_get_vq (m2m_ctx,V4L2_BUF_TYPE_VIDEO_OUTPUT ); +} +EXPORT_SYMBOL (v4l2_m2m_get_src_vq); + +/** +* v4l2_m2m_get_dst_vq() - return videobuf_queue for destination buffers +*/ +struct videobuf_queue *v4l2_m2m_get_dst_vq (struct v4l2_m2m_ctx *m2m_ctx) +{ +return v4l2_m2m_get_vq (m2m_ctx,V4L2_BUF_TYPE_VIDEO_CAPTURE ); +} +EXPORT_SYMBOL (v4l2_m2m_get_dst_vq); + +/** +* v4l2_m2m_next_buf() - return next buffer from the list of ready buffers +*/ +static void *v4l2_m2m_next_buf (struct v4l2_m2m_ctx *m2m_ctx, +enum v4l2_buf_type type) +{
drivers/media/video/v4l2-mem2mem.c line 120
Quoted 10 lines
drivers/media/video/v4l2-mem2mem.c new file +10
-0+struct v4l2_m2m_queue_ctx *q_ctx; +struct videobuf_buffer *vb =NULL ; + + q_ctx =get_queue_ctx (m2m_ctx, type); + + vb = list_entry(q_ctx->rdy_queue.next, struct videobuf_buffer, queue); + vb->state =VIDEOBUF_ACTIVE ; + +return vb; +}
I have some questions/concerns on this:
1. There is no protection on access to q_ctx->rdy_queue.next and this function can be indirectly called from outside this file. In other places "rdy_queue.next" access is protected with the q.irqlock. I know on x86 all reads are atomic, so just reading the "next" is probably safe, but I suspect you won't be using x86.
2. There is no check if the list is empty. In the case of an empty list, "next" will actually point back to the list head, and vb will not point to a valid video_buffer object. The vb->state assignment will end up corrupting something.
Maybe I'm missing something.
Quoted 31 lines
drivers/media/video/v4l2-mem2mem.c new file +31
-0+/** +* v4l2_m2m_next_src_buf() - return next source buffer from the list of ready +* buffers +*/ +inline void *v4l2_m2m_next_src_buf (struct v4l2_m2m_ctx *m2m_ctx) +{ +return v4l2_m2m_next_buf (m2m_ctx,V4L2_BUF_TYPE_VIDEO_OUTPUT ); +} +EXPORT_SYMBOL (v4l2_m2m_next_src_buf); + +/** +* v4l2_m2m_next_dst_buf() - return next destination buffer from the list of +* ready buffers +*/ +inline void *v4l2_m2m_next_dst_buf (struct v4l2_m2m_ctx *m2m_ctx) +{ +return v4l2_m2m_next_buf (m2m_ctx,V4L2_BUF_TYPE_VIDEO_CAPTURE ); +} +EXPORT_SYMBOL (v4l2_m2m_next_dst_buf); + +/** +* v4l2_m2m_buf_remove() - take off a buffer from the list of ready buffers and +* return it +*/ +static void *v4l2_m2m_buf_remove (struct v4l2_m2m_ctx *m2m_ctx, +enum v4l2_buf_type type) +{ +struct v4l2_m2m_queue_ctx *q_ctx; +struct videobuf_buffer *vb =NULL ; +unsigned long flags =0 ; +
drivers/media/video/v4l2-mem2mem.c line 162
Quoted 10 lines
drivers/media/video/v4l2-mem2mem.c new file +10
-0+ q_ctx =get_queue_ctx (m2m_ctx, type); + +spin_lock_irqsave (q_ctx->q.irqlock, flags); + vb = list_entry(q_ctx->rdy_queue.next, struct videobuf_buffer, queue); +list_del (&vb->queue); + q_ctx->num_rdy--; +spin_unlock_irqrestore (q_ctx->q.irqlock, flags); + +return vb; +}
Again, there is no check for an empty list here, and it can be indirectly called by callers external to this file.
Quoted 28 lines
drivers/media/video/v4l2-mem2mem.c new file +28
-0+/** +* v4l2_m2m_src_buf_remove() - take off a srouce buffer from the list of ready +* buffers and return it +*/ +void *v4l2_m2m_src_buf_remove (struct v4l2_m2m_ctx *m2m_ctx) +{ +return v4l2_m2m_buf_remove (m2m_ctx,V4L2_BUF_TYPE_VIDEO_OUTPUT ); +} +EXPORT_SYMBOL (v4l2_m2m_src_buf_remove); + +/** +* v4l2_m2m_dst_buf_remove() - take off a destination buffer from the list of +* ready buffers and return it +*/ +void *v4l2_m2m_dst_buf_remove (struct v4l2_m2m_ctx *m2m_ctx) +{ +return v4l2_m2m_buf_remove (m2m_ctx,V4L2_BUF_TYPE_VIDEO_CAPTURE ); + +} +EXPORT_SYMBOL (v4l2_m2m_dst_buf_remove); + + +/* +* Scheduling handlers +*/ + +/** +* v4l2_m2m_get_curr_priv() - return driver private data for the currently
drivers/media/video/v4l2-mem2mem.c line 201
Quoted 10 lines
drivers/media/video/v4l2-mem2mem.c new file +10
-0+* running instance or NULL if no instance is running +*/ +void *v4l2_m2m_get_curr_priv (struct v4l2_m2m_dev *m2m_dev) +{ +if (!m2m_dev->curr_ctx) +return NULL ; +else +return m2m_dev->curr_ctx->priv; +} +EXPORT_SYMBOL (v4l2_m2m_get_curr_priv);
Every other access to m2m_dev->curr_ctx is protected by the spin lock, but for this externally callable function it is not. The curr_ctx could be set back to NULL between the if() and the return in the else clause.
Quoted 31 lines
drivers/media/video/v4l2-mem2mem.c new file +31
-0+ +/** +* v4l2_m2m_try_run() - select next job to perform and run it if possible +* +* Get next transaction (if present) from the waiting jobs list and run it. +*/ +static void v4l2_m2m_try_run (struct v4l2_m2m_dev *m2m_dev) +{ +unsigned long flags =0 ; + +spin_lock_irqsave (&m2m_dev->job_spinlock, flags); +if (NULL != m2m_dev->curr_ctx) { +spin_unlock_irqrestore (&m2m_dev->job_spinlock, flags); +dprintk ("Another instance is running, won't run now\n" ); +return ; + } + +if (list_empty (&m2m_dev->jobqueue)) { +spin_unlock_irqrestore (&m2m_dev->job_spinlock, flags); +dprintk ("No job pending\n" ); +return ; + } + + m2m_dev->curr_ctx =list_entry (m2m_dev->jobqueue.next, + struct v4l2_m2m_ctx, queue); + m2m_dev->curr_ctx->job_flags |=TRANS_RUNNING ; +spin_unlock_irqrestore (&m2m_dev->job_spinlock, flags); + + m2m_dev->m2m_ops->device_run (m2m_dev->curr_ctx->priv); + +return ;
drivers/media/video/v4l2-mem2mem.c line 242
Quoted 10 lines
drivers/media/video/v4l2-mem2mem.c new file +10
-0+} + +/** +* v4l2_m2m_schedule() - add an instance to the pending job queue +* @m2m_ctx: The instance to be added to the pending job queue +* +* Called when an instance is fully prepared to run a transaction, i.e. the +* instance will not sleep before finishing the transaction when run. +* If an instance is already on the queue, it will not be added for the second +* time and it is the responsibility of the instance to retry this at a later
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Quoted 6 lines
drivers/media/video/v4l2-mem2mem.c new file +6
-0+* time. +*/ +static void v4l2_m2m_schedule (struct v4l2_m2m_ctx *m2m_ctx) +{ +struct v4l2_m2m_dev *dev = m2m_ctx->m2m_dev; +unsigned long flags =0 ;
drivers/media/video/v4l2-mem2mem.c line 258
Quoted 10 lines
drivers/media/video/v4l2-mem2mem.c new file +10
-0+ +spin_lock_irqsave (&dev->job_spinlock, flags); +if (!(m2m_ctx->job_flags &TRANS_QUEUED )) { +list_add_tail (&m2m_ctx->queue, &dev->jobqueue); + m2m_ctx->job_flags |=TRANS_QUEUED ; + } +spin_unlock_irqrestore (&dev->job_spinlock, flags); + +v4l2_m2m_try_run (dev); +}
So how is the calling instance supposed to know to retry when the function returns void?
Quoted 46 lines
drivers/media/video/v4l2-mem2mem.c new file +46
-0+/** +* v4l2_m2m_try_schedule() - check whether an instance is ready to be added to +* the pending job queue and add it if so. +* @m2m_ctx: m2m context assigned to the instance to be checked +* +* There are three basic requirements an instance has to meet to be able to run: +* 1) at least one source buffer has to be queued, +* 2) at least one destination buffer has to be queued, +* 3) streaming has to be on. +* +* There can also be additional, custom requirements. In such case the driver +* should supply a custom method (job_ready in v4l2_m2m_ops) that should +* return 1 * if the instance is ready. +* An example of the above could be an instance that requires more than one +* src/dst buffer per transaction. +*/ +static void v4l2_m2m_try_schedule (struct v4l2_m2m_ctx *m2m_ctx) +{ +struct v4l2_m2m_dev *m2m_dev; +unsigned long flags =0 ; + + m2m_dev = m2m_ctx->m2m_dev; +dprintk ("Trying to schedule a job for m2m_ctx: %p\n" , m2m_ctx); + +spin_lock_irqsave (&m2m_dev->job_spinlock, flags); +if (m2m_ctx->job_flags &TRANS_QUEUED ) { +spin_unlock_irqrestore (&m2m_dev->job_spinlock, flags); +dprintk ("On job queue already\n" ); +return ; + } +spin_unlock_irqrestore (&m2m_dev->job_spinlock, flags); + +/* Checking only one queue is enough, we always turn on both */ +if (!m2m_ctx->out_q_ctx.q.streaming) { +dprintk ("Streaming not on, will not schedule\n" ); +return ; + } + +if (list_empty (&m2m_ctx->out_q_ctx.rdy_queue)) { +dprintk ("No input buffers available\n" ); +return ; + } +if (list_empty (&m2m_ctx->cap_q_ctx.rdy_queue)) { +dprintk ("No output buffers available\n" ); +return ; + }
drivers/media/video/v4l2-mem2mem.c line 315
Quoted 10 lines
drivers/media/video/v4l2-mem2mem.c new file +10
-0+ +if (m2m_dev->m2m_ops->job_ready + && (!m2m_dev->m2m_ops->job_ready (m2m_ctx->priv))) { +dprintk ("Driver not ready\n" ); +return ; + } + +dprintk ("Instance ready to be scheduled\n" ); +v4l2_m2m_schedule (m2m_ctx); +}
Again, accesss to the rdy_queue objects are not protected.
Since you have a num_ready variable, how about changing that from a u8 to an atomic_t and checking num_ready for when you need to check if the rdy_queue is empty or not?
Quoted 14 lines
drivers/media/video/v4l2-mem2mem.c new file +14
-0+/** +* v4l2_m2m_job_finish() - inform the framework that a job has been finished +* and have it clean up +* +* Called by a driver to yield back the device after it has finished with it. +* Should be called as soon as possible after reaching a state which allows +* other instances to take control of the device. +* +* TODO: An instance that fails to give back the device before a predefined +* amount of time may have its device ownership taken away forcibly. +*/ +void v4l2_m2m_job_finish (struct v4l2_m2m_dev *m2m_dev, +struct v4l2_m2m_ctx *m2m_ctx) +{
drivers/media/video/v4l2-mem2mem.c line 340
Quoted 10 lines
drivers/media/video/v4l2-mem2mem.c new file +10
-0+unsigned long flags =0 ; + +spin_lock_irqsave (&m2m_dev->job_spinlock, flags); +if (!m2m_dev->curr_ctx || m2m_dev->curr_ctx != m2m_ctx) { +spin_unlock_irqrestore (&m2m_dev->job_spinlock, flags); +dprintk ("Called by an instance not currently running\n" ); +return ; + } + +/*mutex_lock(&m2m_dev->dev_mutex);*/
This looks like it should be deleted: m2m_dev doesn't have a dev_mutex member. I don't think you would want to sleep while holding a spinlock anyway.
Quoted 106 lines
drivers/media/video/v4l2-mem2mem.c new file +106
-0+list_del (&m2m_dev->curr_ctx->queue); + m2m_dev->curr_ctx->job_flags &= ~(TRANS_QUEUED |TRANS_RUNNING ); + m2m_dev->curr_ctx =NULL ; + +spin_unlock_irqrestore (&m2m_dev->job_spinlock, flags); + +v4l2_m2m_try_run (m2m_dev); +} +EXPORT_SYMBOL (v4l2_m2m_job_finish); + +/** +* v4l2_m2m_reqbufs() - multi-queue-aware REQBUFS multiplexer +*/ +int v4l2_m2m_reqbufs (struct file *file,struct v4l2_m2m_ctx *m2m_ctx, +struct v4l2_requestbuffers *reqbufs) +{ +struct videobuf_queue *vq; + + vq =v4l2_m2m_get_vq (m2m_ctx, reqbufs->type); +return videobuf_reqbufs (vq, reqbufs); +} +EXPORT_SYMBOL_GPL (v4l2_m2m_reqbufs); + +/** +* v4l2_m2m_querybuf() - multi-queue-aware QUERYBUF multiplexer +* +* See v4l2_m2m_mmap() documentation for details. +*/ +int v4l2_m2m_querybuf (struct file *file,struct v4l2_m2m_ctx *m2m_ctx, +struct v4l2_buffer *buf) +{ +struct videobuf_queue *vq; +int ret =0 ; + + vq =v4l2_m2m_get_vq (m2m_ctx, buf->type); + ret =videobuf_querybuf (vq, buf); + +if (buf->memory ==V4L2_MEMORY_MMAP + && vq->type ==V4L2_BUF_TYPE_VIDEO_CAPTURE ) { + buf->m.offset +=DST_QUEUE_OFF_BASE ; + } + +return ret; +} +EXPORT_SYMBOL_GPL (v4l2_m2m_querybuf); + +/** +* v4l2_m2m_qbuf() - enqueue a source or destination buffer, depending on +* the type +*/ +int v4l2_m2m_qbuf (struct file *file,struct v4l2_m2m_ctx *m2m_ctx, +struct v4l2_buffer *buf) +{ +struct videobuf_queue *vq; + + vq =v4l2_m2m_get_vq (m2m_ctx, buf->type); +return videobuf_qbuf (vq, buf); +} +EXPORT_SYMBOL_GPL (v4l2_m2m_qbuf); + +/** +* v4l2_m2m_dqbuf() - dequeue a source or destination buffer, depending on +* the type +*/ +int v4l2_m2m_dqbuf (struct file *file,struct v4l2_m2m_ctx *m2m_ctx, +struct v4l2_buffer *buf) +{ +struct videobuf_queue *vq; + + vq =v4l2_m2m_get_vq (m2m_ctx, buf->type); +return videobuf_dqbuf (vq, buf, file->f_flags &O_NONBLOCK ); +} +EXPORT_SYMBOL_GPL (v4l2_m2m_dqbuf); + +/** +* v4l2_M2m_streamon() - start streaming +*/ +int v4l2_m2m_streamon (struct file *file,struct v4l2_m2m_ctx *m2m_ctx, +enum v4l2_buf_type type) +{ +int ret =0 ; + +/* These can fail only if the queues are in use, +* but they shouldn't be as we are managing instances manually */ + ret =videobuf_streamon (&m2m_ctx->out_q_ctx.q); +if (ret) { +printk (KERN_ERR "Streamon on output queue failed\n" ); +return ret; + } + + ret =videobuf_streamon (&m2m_ctx->cap_q_ctx.q); +if (ret) { +printk (KERN_ERR "Streamon on capture queue failed\n" ); +return ret; + } + +v4l2_m2m_try_schedule (m2m_ctx); + +return 0 ; +} +EXPORT_SYMBOL_GPL (v4l2_m2m_streamon); + +/** +* v4l2_m2m_streamoff() - stop streaming +*/ +int v4l2_m2m_streamoff(struct file *file, struct v4l2_m2m_ctx *m2m_ctx,
drivers/media/video/v4l2-mem2mem.c line 456
Quoted 10 lines
drivers/media/video/v4l2-mem2mem.c new file +10
-0+ enum v4l2_buf_type type) +{ +/* streamoff() fails only when we are not streaming */ +if (videobuf_streamoff (&m2m_ctx->out_q_ctx.q) + ||videobuf_streamoff (&m2m_ctx->cap_q_ctx.q)) +return -EINVAL ; + +return 0 ; +} +EXPORT_SYMBOL_GPL (v4l2_m2m_streamoff);
The second videobuf_streamoff() won't get called if the first one fails. I don't know if that was intentional or not. It probably doesn't matter.
Quoted 186 lines
drivers/media/video/v4l2-mem2mem.c new file +186
-0+/** +* v4l2_m2m_poll() - poll replacement, for destination buffers only +* +* Call from driver's poll() function. Will poll the destination queue only. +*/ +unsigned int v4l2_m2m_poll (struct file *file,struct v4l2_m2m_ctx *m2m_ctx, +struct poll_table_struct *wait) +{ +struct videobuf_queue *dst_q =NULL ; +struct videobuf_buffer *vb =NULL ; +unsigned int rc =0 ; + + dst_q =v4l2_m2m_get_dst_vq (m2m_ctx); + +mutex_lock (&dst_q->vb_lock); + +if (dst_q->streaming) { +if (!list_empty (&dst_q->stream)) + vb =list_entry (dst_q->stream.next, + struct videobuf_buffer, stream); + } + +if (!vb) + rc =POLLERR ; + +if (0 == rc) { +poll_wait (file, &vb->done, wait); +if (vb->state ==VIDEOBUF_DONE || vb->state ==VIDEOBUF_ERROR ) + rc =POLLOUT |POLLRDNORM ; + } + +mutex_unlock (&dst_q->vb_lock); +return rc; +} +EXPORT_SYMBOL_GPL (v4l2_m2m_poll); + +/** +* v4l2_m2m_mmap() - source and destination queues-aware mmap multiplexer +* +* Call from driver's mmap() function. Will handle mmap() for both queues +* seamlessly for videobuffer, which will receive normal per-queue offsets and +* proper videobuf queue pointers. The differentation is made outside videobuf +* by adding a predefined offset to buffers from one of the queues and +* subtracting it before passing it back to videobuf. Only drivers (and +* thus applications) receive modified offsets. +*/ +int v4l2_m2m_mmap (struct file *file,struct v4l2_m2m_ctx *m2m_ctx, +struct vm_area_struct *vma) +{ +unsigned long offset = vma->vm_pgoff <<PAGE_SHIFT ; +struct videobuf_queue *vq; + +if (offset <DST_QUEUE_OFF_BASE ) { + vq =v4l2_m2m_get_src_vq (m2m_ctx); + }else { + vq =v4l2_m2m_get_dst_vq (m2m_ctx); + vma->vm_pgoff -= (DST_QUEUE_OFF_BASE >>PAGE_SHIFT ); + } + +return videobuf_mmap_mapper (vq, vma); +} +EXPORT_SYMBOL (v4l2_m2m_mmap); + +/** +* v4l2_m2m_init() - initialize per-driver m2m data +* +* Usually called from driver's probe() function. +*/ +struct v4l2_m2m_dev *v4l2_m2m_init (struct v4l2_m2m_ops *m2m_ops) +{ +struct v4l2_m2m_dev *m2m_dev; + +if (!m2m_ops) +return ERR_PTR (-EINVAL ); + +/*BUG_ON(!m2m_ops->job_ready);*/ +BUG_ON (!m2m_ops->device_run); +BUG_ON (!m2m_ops->job_abort); + + m2m_dev =kzalloc (sizeof *m2m_dev,GFP_KERNEL ); +if (!m2m_dev) +return ERR_PTR (-ENOMEM ); + + m2m_dev->curr_ctx =NULL ; + m2m_dev->m2m_ops = m2m_ops; +INIT_LIST_HEAD (&m2m_dev->jobqueue); +spin_lock_init (&m2m_dev->job_spinlock); + +return m2m_dev; +} +EXPORT_SYMBOL_GPL (v4l2_m2m_init); + +/** +* v4l2_m2m_release() - cleans up and frees a m2m_dev structure +* +* Usually called from driver's remove() function. +*/ +void v4l2_m2m_release (struct v4l2_m2m_dev *m2m_dev) +{ +kfree (m2m_dev); +} +EXPORT_SYMBOL_GPL (v4l2_m2m_release); + +/** +* v4l2_m2m_ctx_init() - allocate and initialize a m2m context +* @priv - driver's instance private data +* @m2m_dev - a previously initialized m2m_dev struct +* @vq_init - a callback for queue type-specific initialization function to be +* used for initializing videobuf_queues +* +* Usually called from driver's open() function. +*/ +struct v4l2_m2m_ctx *v4l2_m2m_ctx_init (void *priv,struct v4l2_m2m_dev *m2m_dev, +void (*vq_init)(void *priv,struct videobuf_queue *, +enum v4l2_buf_type )) +{ +struct v4l2_m2m_ctx *m2m_ctx; +struct v4l2_m2m_queue_ctx *out_q_ctx; +struct v4l2_m2m_queue_ctx *cap_q_ctx; + +if (!vq_init) +return ERR_PTR (-EINVAL ); + + m2m_ctx =kzalloc (sizeof *m2m_ctx,GFP_KERNEL ); +if (!m2m_ctx) +return ERR_PTR (-ENOMEM ); + + m2m_ctx->priv = priv; + m2m_ctx->m2m_dev = m2m_dev; + + out_q_ctx =get_queue_ctx (m2m_ctx,V4L2_BUF_TYPE_VIDEO_OUTPUT ); + cap_q_ctx =get_queue_ctx (m2m_ctx,V4L2_BUF_TYPE_VIDEO_CAPTURE ); + +INIT_LIST_HEAD (&out_q_ctx->rdy_queue); +INIT_LIST_HEAD (&cap_q_ctx->rdy_queue); + +/*spin_lock_init(&m2m_ctx->queue_lock);*/ +INIT_LIST_HEAD (&m2m_ctx->queue); + +vq_init (priv, &out_q_ctx->q,V4L2_BUF_TYPE_VIDEO_OUTPUT ); +vq_init (priv, &cap_q_ctx->q,V4L2_BUF_TYPE_VIDEO_CAPTURE ); + out_q_ctx->q.priv_data = cap_q_ctx->q.priv_data = priv; + +return m2m_ctx; +} +EXPORT_SYMBOL_GPL (v4l2_m2m_ctx_init); + +/** +* v4l2_m2m_ctx_release() - release m2m context +* +* Usually called from driver's release() function. +*/ +void v4l2_m2m_ctx_release (struct v4l2_m2m_ctx *m2m_ctx) +{ +struct v4l2_m2m_dev *m2m_dev; +struct videobuf_buffer *vb; +unsigned long flags =0 ; + + m2m_dev = m2m_ctx->m2m_dev; + +spin_lock_irqsave (&m2m_dev->job_spinlock, flags); +if (m2m_ctx->job_flags &TRANS_RUNNING ) { +spin_unlock_irqrestore (&m2m_dev->job_spinlock, flags); + m2m_dev->m2m_ops->job_abort (m2m_ctx->priv); +dprintk ("m2m_ctx %p running, will wait to complete" , m2m_ctx); + vb =v4l2_m2m_next_dst_buf (m2m_ctx); +BUG_ON (NULL == vb); +wait_event (vb->done, vb->state !=VIDEOBUF_ACTIVE + && vb->state !=VIDEOBUF_QUEUED ); + }else if (m2m_ctx->job_flags &TRANS_QUEUED ) { +list_del (&m2m_ctx->queue); + m2m_ctx->job_flags &= ~(TRANS_QUEUED |TRANS_RUNNING ); +spin_unlock_irqrestore (&m2m_dev->job_spinlock, flags); +dprintk ("m2m_ctx: %p had been on queue and was removed\n" , + m2m_ctx); + }else { +/* Do nothing, was not on queue/running */ +spin_unlock_irqrestore (&m2m_dev->job_spinlock, flags); + } + +videobuf_stop (&m2m_ctx->cap_q_ctx.q); +videobuf_stop (&m2m_ctx->out_q_ctx.q); + +videobuf_mmap_free (&m2m_ctx->cap_q_ctx.q); +videobuf_mmap_free (&m2m_ctx->out_q_ctx.q); +
drivers/media/video/v4l2-mem2mem.c line 653
Quoted 10 lines
drivers/media/video/v4l2-mem2mem.c new file +10
-0+kfree (m2m_ctx); +} +EXPORT_SYMBOL_GPL (v4l2_m2m_ctx_release); + +/** +* v4l2_m2m_buf_queue() - add a buffer to the proper ready buffers list. +* +* Call from withing buf_queue() videobuf_queue_ops callback. +*/ +/* Locking: Caller holds q->irqlock */
If that's a condition that applies to calling other functions that access the rdy_queue then that needs to be indicated in their comments too.
That's all for now.
Regards,
Andy
Quoted 177 lines
drivers/media/video/v4l2-mem2mem.c new file +18
-0+void v4l2_m2m_buf_queue (struct v4l2_m2m_ctx *m2m_ctx,struct videobuf_queue *vq, +struct videobuf_buffer *vb) +{ +struct v4l2_m2m_queue_ctx *q_ctx; + + q_ctx =get_queue_ctx (m2m_ctx, vq->type); +if (!q_ctx) +return ; + +list_add_tail (&vb->queue, &q_ctx->rdy_queue); + q_ctx->num_rdy++; + + vb->state =VIDEOBUF_QUEUED ; + +v4l2_m2m_try_schedule (m2m_ctx); +} +EXPORT_SYMBOL_GPL (v4l2_m2m_buf_queue); +include/media/v4l2-mem2mem.h new file +153
-0@@ -0,0 +1,153 @@ +/* +* Memory-to-memory device framework for Video for Linux 2. +* +* Helper functions for devices that use memory buffers for both source +* and destination. +* +* Copyright (c) 2009 Samsung Electronics Co., Ltd. +* Pawel Osciak, <[email protected]> +* Marek Szyprowski, <[email protected]> +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by the +* Free Software Foundation; either version 2 of the +* License, or (at your option) any later version +*/ + +#ifndef _MEDIA_V4L2_MEM2MEM_H +#define _MEDIA_V4L2_MEM2MEM_H + +#include <media/videobuf-core.h> + +/** +* struct v4l2_m2m_ops - mem-to-mem device driver callbacks +* @device_run: required. Begin the actual job (transaction) inside this +* callback. +* The job does NOT have to end before this callback returns +* (and it will be the usual case). When the job finishes, +* v4l2_m2m_job_finish() has to be called. +* @job_ready: optional. Should return 0 if the driver does not have a job +* fully prepared to run yet (i.e. it will not be able to finish a +* transactio without sleeping). If not provided, it will be +* assumed that one source and one destination buffer are all +* that is required for the driver to perform one full transaction. +* @job_abort: required. Informs the driver that it has to abort the currently +* running transaction as soon as possible (i.e. as soon as it can +* stop the device safely; e.g. in the next interrupt handler), +* even if the transaction would not have been finished by then. +* After the driver performs the necessary steps, it has to call +* v4l2_m2m_job_finish() (as if the transaction ended normally). +* This function does not have to (and will usually not) wait +* until the device enters a state when it can be stopped. +*/ +struct v4l2_m2m_ops { +void (*device_run)(void *priv); +int (*job_ready)(void *priv); +void (*job_abort)(void *priv); +}; + +struct v4l2_m2m_dev ; + +struct v4l2_m2m_queue_ctx { +/* private: internal use only */ +struct videobuf_queue q; + +/* Base value for offsets of mmaped buffers on this queue */ +unsigned long offset_base; + +/* Queue for buffers ready to be processed as soon as this +* instance receives access to the device */ +struct list_head rdy_queue; +u8 num_rdy; +}; + +struct v4l2_m2m_ctx { +/* private: internal use only */ +struct v4l2_m2m_dev *m2m_dev; + +/* Capture (output to memory) queue context */ +struct v4l2_m2m_queue_ctx cap_q_ctx; + +/* Output (input from memory) queue context */ +struct v4l2_m2m_queue_ctx out_q_ctx; + +/* For device job queue */ +struct list_head queue; +unsigned long job_flags; + +/* Instance private data */ +void *priv; +}; + +void *v4l2_m2m_get_curr_priv (struct v4l2_m2m_dev *m2m_dev); + +struct videobuf_queue *v4l2_m2m_get_src_vq (struct v4l2_m2m_ctx *m2m_ctx); +struct videobuf_queue *v4l2_m2m_get_dst_vq (struct v4l2_m2m_ctx *m2m_ctx); +struct videobuf_queue *v4l2_m2m_get_vq (struct v4l2_m2m_ctx *m2m_ctx, +enum v4l2_buf_type type); + +void v4l2_m2m_job_finish (struct v4l2_m2m_dev *m2m_dev, +struct v4l2_m2m_ctx *m2m_ctx); + +void *v4l2_m2m_next_src_buf (struct v4l2_m2m_ctx *m2m_ctx); +void *v4l2_m2m_next_dst_buf (struct v4l2_m2m_ctx *m2m_ctx); + +void *v4l2_m2m_src_buf_remove (struct v4l2_m2m_ctx *m2m_ctx); +void *v4l2_m2m_dst_buf_remove (struct v4l2_m2m_ctx *m2m_ctx); + + +int v4l2_m2m_reqbufs (struct file *file,struct v4l2_m2m_ctx *m2m_ctx, +struct v4l2_requestbuffers *reqbufs); + +int v4l2_m2m_querybuf (struct file *file,struct v4l2_m2m_ctx *m2m_ctx, +struct v4l2_buffer *buf); + +int v4l2_m2m_qbuf (struct file *file,struct v4l2_m2m_ctx *m2m_ctx, +struct v4l2_buffer *buf); +int v4l2_m2m_dqbuf (struct file *file,struct v4l2_m2m_ctx *m2m_ctx, +struct v4l2_buffer *buf); + +int v4l2_m2m_streamon (struct file *file,struct v4l2_m2m_ctx *m2m_ctx, +enum v4l2_buf_type type); +int v4l2_m2m_streamoff (struct file *file,struct v4l2_m2m_ctx *m2m_ctx, +enum v4l2_buf_type type); + +unsigned int v4l2_m2m_poll (struct file *file,struct v4l2_m2m_ctx *m2m_ctx, +struct poll_table_struct *wait); + +int v4l2_m2m_mmap (struct file *file,struct v4l2_m2m_ctx *m2m_ctx, +struct vm_area_struct *vma); + +struct v4l2_m2m_dev *v4l2_m2m_init (struct v4l2_m2m_ops *m2m_ops); +void v4l2_m2m_release (struct v4l2_m2m_dev *m2m_dev); + +struct v4l2_m2m_ctx *v4l2_m2m_ctx_init (void *priv,struct v4l2_m2m_dev *m2m_dev, +void (*vq_init)(void *priv,struct videobuf_queue *, +enum v4l2_buf_type )); +void v4l2_m2m_ctx_release (struct v4l2_m2m_ctx *m2m_ctx); + +void v4l2_m2m_buf_queue (struct v4l2_m2m_ctx *m2m_ctx,struct videobuf_queue *vq, +struct videobuf_buffer *vb); + +/** +* v4l2_m2m_num_src_bufs_ready() - return the number of source buffers ready for +* use +*/ +static inline +unsigned int v4l2_m2m_num_src_bufs_ready (struct v4l2_m2m_ctx *m2m_ctx) +{ +return m2m_ctx->cap_q_ctx.num_rdy; +} + +/** +* v4l2_m2m_num_src_bufs_ready() - return the number of destination buffers +* ready for use +*/ +static inline +unsigned int v4l2_m2m_num_dst_bufs_ready (struct v4l2_m2m_ctx *m2m_ctx) +{ +return m2m_ctx->out_q_ctx.num_rdy; +} + +#endif /* _MEDIA_V4L2_MEM2MEM_H */ +
PATCH 2/2 V4L: Add a mem-to-mem V4L2 framework test device.
In reply to Pawel OsciakThis is a virtual device driver for testing the mem-to-mem V4L2 framework. It simulates a device that uses memory buffers for both source and destination, processes the data and issues an "IRQ" (simulated by a timer). The device is capable of multi-instance, multi-buffer-per-transaction operation (via the mem2mem framework).
--- drivers/media/video/Kconfig | 14 + drivers/media/video/Makefile | 1 + drivers/media/video/mem2mem_testdev.c | 1052 +++++++++++++++++++++++++++++++++ 3 files changed, 1067 insertions(+), 0 deletions(-) create mode 100644 drivers/media/video/mem2mem_testdev.c
drivers/media/video/Kconfig +14 -0
@@ -1089,3 +1089,17 @@ menuconfig V4L_MEM2MEM_DRIVERS use system memory for both source and destination buffers, as opposed to capture and output drivers, which use memory buffers for just one of those. + +if V4L_MEM2MEM_DRIVERS + +config VIDEO_MEM2MEM_TESTDEV + tristate "Virtual test device for mem2mem framework" + depends on VIDEO_DEV && VIDEO_V4L2 + select VIDEOBUF_VMALLOC + select V4L2_MEM2MEM_DEV + default n + ---help--- + This is a virtual test device for the memory-to-memory driver + framework. + +endif # V4L_MEM2MEM_DRIVERS
drivers/media/video/Makefile +1 -0
@@ -149,6 +149,7 @@ obj-$(CONFIG_VIDEO_IVTV) += ivtv/ obj-$(CONFIG_VIDEO_CX18) += cx18/ obj-$(CONFIG_VIDEO_VIVI) += vivi.o +obj-$(CONFIG_VIDEO_MEM2MEM_TESTDEV) += mem2mem_testdev.o obj-$(CONFIG_VIDEO_CX23885) += cx23885/ obj-$(CONFIG_VIDEO_OMAP2) += omap2cam.o
drivers/media/video/mem2mem_testdev.c new file +1052 -0
1,052 lines not on this page · raw message
Signature · 1 line
1.6.4.2.253.g0b1fac
PATCH v2 1/1 Mem-to-mem userspace test application.
In reply to Pawel OsciakThis is an example application for testing mem-to-mem framework using mem2mem-testdev device.
It is intended to be executed multiple times in parallel to test multi-instance operation and scheduling. Each process can be configured differently using command-line arguments.
The application opens video test device and framebuffer, sets up params, queues src/dst buffers and displays processed results on the framebuffer.
Configurable parameters: starting point on the framebuffer, width/height of buffers, transaction length (in buffers), transaction duration, total number of frames to be processed.
Tested on a 800x480 framebuffer with the following script:
#!/bin/bash
for i in {0..3}
do
((x=$i * 100))
./process-vmalloc 0 $(($i + 1)) $((2000 - $i * 500)) $((($i+1) * 4)) \
$x $x 100 100 &done
---
process-vmalloc.c new file +420 -0
@@ -0,0 +1,420 @@ +/** +* process-vmalloc.c +* Capture+output (process) V4L2 device tester. +* +* Pawel Osciak, p.osciak at samsung.com +* 2009, Samsung Electronics Co., Ltd. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by the +* Free Software Foundation; either version 2 of the License, or (at your +* option) any later version +*/ + +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <assert.h> +#include <time.h> +#include <errno.h> + +#include <fcntl.h> +#include <unistd.h> +#include <sys/ioctl.h> +#include <sys/types.h> +#include <stdint.h> + +#include <linux/fb.h> +#include <linux/videodev2.h> + +#include <sys/mman.h> + +#define V4L2_CID_TRANS_TIME_MSEC (V4L2_CID_PRIVATE_BASE) +#define V4L2_CID_TRANS_NUM_BUFS (V4L2_CID_PRIVATE_BASE + 1) + +#define VIDEO_DEV_NAME "/dev/video0" +#define FB_DEV_NAME "/dev/fb0" +#define NUM_BUFS 4 +#define NUM_FRAMES 16 + +#define perror_exit (cond, func)\ + if (cond) {\ + fprintf(stderr, "%s:%d: ", __func__, __LINE__);\ + perror(func);\ + exit(EXIT_FAILURE);\ + } + +#define error_exit (cond, func)\ + if (cond) {\ + fprintf(stderr, "%s:%d: failed\n", func, __LINE__);\ + exit(EXIT_FAILURE);\ + } + +#define perror_ret (cond, func)\ + if (cond) {\ + fprintf(stderr, "%s:%d: ", __func__, __LINE__);\ + perror(func);\ + return ret;\ + } + +#define memzero (x)\ + memset(&(x), 0, sizeof (x)); + +#define PROCESS_DEBUG 1 +#ifdef PROCESS_DEBUG +#define debug (msg, ...)\ + fprintf(stderr, "%s: " msg, __func__, ##__VA_ARGS__); +#else +#define debug (msg, ...) +#endif + +static int vid_fd, fb_fd; +static void *fb_addr; +static char *p_src_buf[NUM_BUFS ], *p_dst_buf[NUM_BUFS ]; +static size_t src_buf_size[NUM_BUFS ], dst_buf_size[NUM_BUFS ]; +static uint32_t num_src_bufs =0 , num_dst_bufs =0 ; + +/* Command-line params */ +int initial_delay =0 ; +int fb_x, fb_y, width, height; +int translen =1 ; +/* For displaying multi-buffer transaction simulations, indicates current +* buffer in an ongoing transaction */ +int curr_buf =0 ; +int transtime =1000 ; +int num_frames =0 ; +off_t fb_off, fb_line_w, fb_buf_w; +struct fb_var_screeninfo fbinfo; + +static void init_video_dev (void ) +{ +int ret; +struct v4l2_capability cap; +struct v4l2_format fmt; +struct v4l2_control ctrl; + + vid_fd =open (VIDEO_DEV_NAME ,O_RDWR |O_NONBLOCK ,0 ); +perror_exit (vid_fd <0 ,"open" ); + + ctrl.id =V4L2_CID_TRANS_TIME_MSEC ; + ctrl.value = transtime; + ret =ioctl (vid_fd,VIDIOC_S_CTRL , &ctrl); +perror_exit (ret !=0 ,"ioctl" ); + + ctrl.id =V4L2_CID_TRANS_NUM_BUFS ; + ctrl.value = translen; + ret =ioctl (vid_fd,VIDIOC_S_CTRL , &ctrl); +perror_exit (ret !=0 ,"ioctl" ); + + ret =ioctl (vid_fd,VIDIOC_QUERYCAP , &cap); +perror_exit (ret !=0 ,"ioctl" ); + +if (!(cap.capabilities &V4L2_CAP_VIDEO_CAPTURE )) { +fprintf (stderr,"Device does not support capture\n" ); +exit (EXIT_FAILURE ); + } +if (!(cap.capabilities &V4L2_CAP_VIDEO_OUTPUT )) { +fprintf (stderr,"Device does not support output\n" ); +exit (EXIT_FAILURE ); + } + +/* Set format for capture */ + fmt.type =V4L2_BUF_TYPE_VIDEO_CAPTURE ; + fmt.fmt.pix.width = width; + fmt.fmt.pix.height = height; + fmt.fmt.pix.pixelformat =V4L2_PIX_FMT_RGB565X ; + fmt.fmt.pix.field =V4L2_FIELD_ANY ; + + ret =ioctl (vid_fd,VIDIOC_S_FMT , &fmt); +perror_exit (ret !=0 ,"ioctl" ); + +/* The same format for output */ + fmt.type =V4L2_BUF_TYPE_VIDEO_OUTPUT ; + fmt.fmt.pix.width = width; + fmt.fmt.pix.height = height; + fmt.fmt.pix.pixelformat =V4L2_PIX_FMT_RGB565X ; + fmt.fmt.pix.field =V4L2_FIELD_ANY ; + + ret =ioctl (vid_fd,VIDIOC_S_FMT , &fmt); +perror_exit (ret !=0 ,"ioctl" ); +} + +static void gen_src_buf (void *p,size_t size) +{ +uint8_t val; + + val =rand () %256 ; +memset (p, val, size); +} + +static void gen_dst_buf (void *p,size_t size) +{ +/* White */ +memset (p,255 ,0 ); +} + +static int read_frame (int last) +{ +struct v4l2_buffer buf; +int ret; +int j; +char * p_fb = fb_addr + fb_off; + +memzero (buf); + + buf.type =V4L2_BUF_TYPE_VIDEO_OUTPUT ; + buf.memory =V4L2_MEMORY_MMAP ; + + ret =ioctl (vid_fd,VIDIOC_DQBUF , &buf); +debug ("Dequeued source buffer, index: %d\n" , buf.index); +if (ret) { +switch (errno) { +case EAGAIN : +debug ("Got EAGAIN\n" ); +return 0 ; + +case EIO : +debug ("Got EIO\n" ); +return 0 ; + +default : +perror ("ioctl" ); +return 0 ; + } + } + +/* Verify we've got a correct buffer */ +assert (buf.index < num_src_bufs); + +/* Enqueue back the buffer (note that the index is preserved) */ +if (!last) { +gen_src_buf (p_src_buf[buf.index], src_buf_size[buf.index]); + buf.type =V4L2_BUF_TYPE_VIDEO_OUTPUT ; + buf.memory =V4L2_MEMORY_MMAP ; + ret =ioctl (vid_fd,VIDIOC_QBUF , &buf); +perror_ret (ret !=0 ,"ioctl" ); + } + + + buf.type =V4L2_BUF_TYPE_VIDEO_CAPTURE ; + +debug ("Dequeuing destination buffer\n" ); + ret =ioctl (vid_fd,VIDIOC_DQBUF , &buf); +if (ret) { +switch (errno) { +case EAGAIN : +debug ("Got EAGAIN\n" ); +return 0 ; + +case EIO : +debug ("Got EIO\n" ); +return 0 ; + +default : +perror ("ioctl" ); +return 1 ; + } + } +debug ("Dequeued dst buffer, index: %d\n" , buf.index); +/* Verify we've got a correct buffer */ +assert (buf.index < num_dst_bufs); + +debug ("Current buffer in the transaction: %d\n" , curr_buf); + p_fb += curr_buf * (height / translen) * fb_line_w; + ++curr_buf; +if (curr_buf >= translen) + curr_buf =0 ; + +/* Display results */ +for (j =0 ; j < height / translen; ++j) { +memcpy (p_fb, (void *)p_dst_buf[buf.index], fb_buf_w); + p_fb += fb_line_w; + } + +/* Enqueue back the buffer */ +if (!last) { +gen_dst_buf (p_dst_buf[buf.index], dst_buf_size[buf.index]); + ret =ioctl (vid_fd,VIDIOC_QBUF , &buf); +perror_ret (ret !=0 ,"ioctl" ); +debug ("Enqueued back dst buffer\n" ); + } + +return 0 ; +} + +void init_usage (int argc,char *argv[]) +{ +if (argc !=9 ) { +printf ("Usage: %s initial_delay bufs_per_transaction " +"trans_length_msec num_frames fb_offset_x fb_offset_y " +"width height\n" , argv[0 ]); +exit (EXIT_FAILURE ); + } + + initial_delay =atoi (argv[1 ]); + translen =atoi (argv[2 ]); + transtime =atoi (argv[3 ]); + num_frames =atoi (argv[4 ]); + fb_x =atoi (argv[5 ]); + fb_y =atoi (argv[6 ]); + width =atoi (argv[7 ]); + height =atoi (argv[8 ]); +debug ("NEW PROCESS: fb_x: %d, fb_y: %d, width: %d, height: %d, " +"translen: %d, transtime: %d, num_frames: %d\n" , + fb_x, fb_y, width, height, translen, transtime, num_frames); +} + +void init_fb (void ) +{ +int ret; +size_t map_size; + + fb_fd =open (FB_DEV_NAME ,O_RDWR ,0 ); +perror_exit (fb_fd <0 ,"open" ); + + ret =ioctl (fb_fd,FBIOGET_VSCREENINFO , &fbinfo); +perror_exit (ret !=0 ,"ioctl" ); +debug ("fbinfo: xres: %d, xres_virt: %d, yres: %d, yres_virt: %d\n" , + fbinfo.xres, fbinfo.xres_virtual, + fbinfo.yres, fbinfo.yres_virtual); + + fb_line_w= fbinfo.xres_virtual * (fbinfo.bits_per_pixel >>3 ); + fb_off = fb_y * fb_line_w + fb_x * (fbinfo.bits_per_pixel >>3 ); + fb_buf_w = width * (fbinfo.bits_per_pixel >>3 ); + map_size = fb_line_w * fbinfo.yres_virtual; + + fb_addr =mmap (0 , map_size,PROT_WRITE |PROT_READ , +MAP_SHARED , fb_fd,0 ); +perror_exit (fb_addr ==MAP_FAILED ,"mmap" ); +} + +int main (int argc,char *argv[]) +{ +int ret =0 ; +int i; +struct v4l2_buffer buf; +struct v4l2_requestbuffers reqbuf; +enum v4l2_buf_type type; +int last =0 ; + +init_usage (argc, argv); +init_fb (); + +srand (time (NULL ) ^getpid ()); +sleep (initial_delay); + +init_video_dev (); + +memzero (reqbuf); + reqbuf.count =NUM_BUFS ; + reqbuf.type =V4L2_BUF_TYPE_VIDEO_OUTPUT ; + type =V4L2_BUF_TYPE_VIDEO_OUTPUT ; + reqbuf.memory =V4L2_MEMORY_MMAP ; + ret =ioctl (vid_fd,VIDIOC_REQBUFS , &reqbuf); +perror_exit (ret !=0 ,"ioctl" ); + num_src_bufs = reqbuf.count; +debug ("Got %d src buffers\n" , num_src_bufs); + + reqbuf.count =NUM_BUFS ; + reqbuf.type =V4L2_BUF_TYPE_VIDEO_CAPTURE ; + type =V4L2_BUF_TYPE_VIDEO_CAPTURE ; + ret =ioctl (vid_fd,VIDIOC_REQBUFS , &reqbuf); +perror_exit (ret !=0 ,"ioctl" ); + num_dst_bufs = reqbuf.count; +debug ("Got %d dst buffers\n" , num_dst_bufs); + +for (i =0 ; i < num_src_bufs; ++i) { + buf.type =V4L2_BUF_TYPE_VIDEO_OUTPUT ; + buf.memory =V4L2_MEMORY_MMAP ; + buf.index = i; + + ret =ioctl (vid_fd,VIDIOC_QUERYBUF , &buf); +perror_exit (ret !=0 ,"ioctl" ); +debug ("QUERYBUF returned offset: %x\n" , buf.m.offset); + + src_buf_size[i] = buf.length; + p_src_buf[i] =mmap (NULL , buf.length, +PROT_READ |PROT_WRITE ,MAP_SHARED , + vid_fd, buf.m.offset); +perror_exit (MAP_FAILED == p_src_buf[i],"mmap" ); + } + +for (i =0 ; i < num_dst_bufs; ++i) { + buf.type =V4L2_BUF_TYPE_VIDEO_CAPTURE ; + buf.memory =V4L2_MEMORY_MMAP ; + buf.index = i; + + ret =ioctl (vid_fd,VIDIOC_QUERYBUF , &buf); +perror_exit (ret !=0 ,"ioctl" ); +debug ("QUERYBUF returned offset: %x\n" , buf.m.offset); + + dst_buf_size[i] = buf.length; + p_dst_buf[i] =mmap (NULL , buf.length, +PROT_READ |PROT_WRITE ,MAP_SHARED , + vid_fd, buf.m.offset); +perror_exit (MAP_FAILED == p_dst_buf[i],"mmap" ); + } + +for (i =0 ; i < num_src_bufs; ++i) { + +gen_src_buf (p_src_buf[i], src_buf_size[i]); + +memzero (buf); + buf.type =V4L2_BUF_TYPE_VIDEO_OUTPUT ; + buf.memory =V4L2_MEMORY_MMAP ; + buf.index = i; + + ret =ioctl (vid_fd,VIDIOC_QBUF , &buf); +perror_exit (ret !=0 ,"ioctl" ); + } + +for (i =0 ; i < num_dst_bufs; ++i) { +memzero (buf); + buf.type =V4L2_BUF_TYPE_VIDEO_CAPTURE ; + buf.memory =V4L2_MEMORY_MMAP ; + buf.index = i; + + ret =ioctl (vid_fd,VIDIOC_QBUF , &buf); +perror_exit (ret !=0 ,"ioctl" ); + } + + ret =ioctl (vid_fd,VIDIOC_STREAMON , &type); +debug ("STREAMON (%d): %d\n" ,VIDIOC_STREAMON , ret); +perror_exit (ret !=0 ,"ioctl" ); + +while (num_frames) { +fd_set read_fds; +int r; + +FD_ZERO (&read_fds); +FD_SET (vid_fd, &read_fds); + +debug ("Before select" ); + r =select (vid_fd +1 , &read_fds,NULL ,NULL ,0 ); +perror_exit (r <0 ,"select" ); +debug ("After select" ); + +if (num_frames ==1 ) + last =1 ; +if (read_frame (last)) { +fprintf (stderr,"Read frame failed\n" ); +break ; + } + --num_frames; +printf ("FRAMES LEFT: %d\n" , num_frames); + } + + +done: +close (vid_fd); +close (fb_fd); + +for (i =0 ; i < num_src_bufs; ++i) +munmap (p_src_buf[i], src_buf_size[i]); + +for (i =0 ; i < num_dst_bufs; ++i) +munmap (p_dst_buf[i], dst_buf_size[i]); + +return ret; +} +
cover Mem-to-mem device framework
In reply to Pawel OsciakQuoted 4 lines · On Wednesday 23 December 2009 14:17:32 Pawel Osciak wrote:
Hello,
this is the second version of the proposed implementation for mem-to-mem memory device framework. Your comments are very welcome.
Hi Pawel,
Thank you for working on this! It's much appreciated. Now I've noticed that patches regarding memory-to-memory and memory pool tend to get very few comments. I suspect that the main reason is that these are SoC-specific features that do not occur in consumer-type products. So most v4l developers do not have the interest and motivation (and time!) to look into this.
I'm CC-ing this reply to developers from Intel, TI, Nokia and Renesas in the hope that they will find some time to review and think about this since this will affect all of them.
One thing that I am missing is a high-level overview of what we want. Currently there are patches/RFCs floating around for memory-to-memory support, multiplanar support and memory-pool support.
What I would like to see is a RFC that ties this all together from the point of view of the public API. I.e. what are the requirements? Possibly solutions? Open questions? Forget about how to implement it for the moment, that will follow from the chosen solutions.
Note that I would suggest though that the memory-pool part is split into two
parts: how to actually allocate the memory is pretty much separate from how v4l will use it. The actual allocation part is probably quite complex and might even be hardware dependent and should be discussed separately. But how to use it is something that can be discussed without needing to know how it was allocated.
The lack of discussion in this area does worry me a bit. IMHO this is a very important area that needs a lot more work. The initiative should be with the SoC companies and right now it seems only Samsung is active.
BTW, what is the status of the multiplanar RFC? I later realized that that RFC might be very useful for adding meta-data to buffers. There are several cases where that is useful: sensors that provide meta-data when capturing a frame and imagepipelines (particularly in memory-to-memory cases) that want to have all parameters as part of the meta-data associated with the image. There may well be more of those.
Regards,
Hans
Quoted 234 lines
In v2.1:
I am very sorry for the resend, but somehow an orphaned endif found its way toKconfigduring the rebase.Changes since v1:
-v4l2_m2m_buf_queue()now requiresm2m_ctxas its argument
-video_queueprivate data stores driver private data
- a new submenu in kconfig for mem-to-mem devices
- minor rebase leftovers cleanupA second patch series followed v2 with a new driver for a real device - Samsung S3C/S5P image rotator, utilizing this framework.
This series contains:
[PATCH v2.1 1/2] V4L: Add memory-to-memory device helper framework for V4L2. [PATCH v2.1 2/2] V4L: Add a mem-to-mem V4L2 framework test device. [EXAMPLE v2] Mem-to-mem userspace test application.
Previous discussion and RFC on this topic:
http://thread.gmane.org/gmane.linux.drivers.video-input-infrastructure/10668A mem-to-mem device is a device that uses memory buffers passed by userspace applications for both source and destination. This is different from existing drivers that use memory buffers for only one of those at once.
In terms of V4L2 such a device would be both of OUTPUT and CAPTURE type. Although no such devices are present in the V4L2 framework, a demand for such a model exists, e.g. for 'resizer devices'.-------------------------------------------------------------------------------
Mem-to-mem devices
-------------------------------------------------------------------------------
In the previous discussion we concluded that we should use one video node with two queues, an output (V4L2_BUF_TYPE_VIDEO_OUTPUT) queue for source buffers and a capture queue (V4L2_BUF_TYPE_VIDEO_CAPTURE) for destination buffers.Each instance has its own set of queues: 2
videobuf_queues, each with a ready buffer queue, managed by the framework. Everything is encapsulated in the queue context struct:struct v4l2_m2m_queue_ctx { struct videobuf_queue q; /* ... */ /* Queue for buffers ready to be processed as soon as this * instance receives access to the device */ struct list_head rdy_queue; /* ... */ };struct v4l2_m2m_ctx { /* ... */ /* Capture (output to memory) queue context */ struct v4l2_m2m_queue_ctx cap_q_ctx; /* Output (input from memory) queue context */ struct v4l2_m2m_queue_ctx out_q_ctx; /* ... */ };Streamon can be called for all instances and will not sleep if another instance is streaming.
vidioc_querycap()should reportV4L2_CAP_VIDEO_CAPTURE|V4L2_CAP_VIDEO_OUTPUT.-------------------------------------------------------------------------------
Queuing and dequeuing buffers
-------------------------------------------------------------------------------
Applications can queue as many buffers as they want and it is not required to queue an equal number of source and destination buffers. If there is not enough buffers of any type, a new transaction will simply not be scheduled.-------------------------------------------------------------------------------
Source and destination formats
-------------------------------------------------------------------------------
Should be set per queue. A helper function to access queues depending on the passed type -v4l2_m2m_get_vq()- is supplied. Most of the format-handling code is normally located in drivers anyway. The only exception is the "field" member of thevideobuf_queuestruct, which has to be set directly. It breaks encapsulation a little bit, but nothing can be done with it.-------------------------------------------------------------------------------
Scheduling
-------------------------------------------------------------------------------
Requirements/assumptions:
1. More than one instance can be open at the same time.
2. Each instance periodically receives exclusive access to the device, performs an operation (operations) and yields back the device in a state that allows other instances to use it.
3. When an instance gets access to the device, it performs a "transaction"/"job". A transaction/job is defined as the shortest operation that cannot/should not be further divided without having to restart it from scratch, or without having to perform expensive reconfiguration of a device, etc.
4. Transactions can use multiple source/destination buffers.
5. Only a driver can tell when it is ready to perform a transaction, so a optional callback is provided for that purpose (job_ready()).There are three common requirements for a transaction to be ready to run:
- at least one source buffer ready
- at least one destination buffer ready
- streaming on
- (optional) driver-specific requirements (driver-specific callback function)So when buffers are queued by
qbuf()or streaming is turned on withstreamon(), the framework callsv4l2_m2m_try_schedule().
v4l2_m2m_try_schedule()
1. Checks for the above conditions.
2. Checks for driver-specific conditions by callingjob_ready()callback, if supplied.
3. If all the checks succeed, it callsv4l2_m2m_schedule()to schedule the transaction.
v4l2_m2m_schedule()
1. Checks whether the transaction is already on job queue and schedules it if not (by adding it to the job queue).
2. Callsv4l2_m2m_try_run().
v4l2_m2m_try_run()
1. Runs a job if and is pending and none is currently running by callingdevice_run()callback.When the
device_run()callback is called, the driver has to begin the transaction. When it is finished, the driver has to callv4l2_m2m_job_finish().
v4l2_m2m_job_finish()
1. Removes the currently running transaction from the job queue and callsv4l2_m2m_try_runto (possibly) run the next pending transaction.There is also support for forced transaction aborting (when an application gets killed). The framework calls
job_abort()callback and the driver has to abort the transaction as soon as possible and callv4l2_m2m_job_finish()to indicate that the transaction has been aborted.Additionally, some kind of timeout for transactions could be added to prevent instances from claiming the device for too long.
-------------------------------------------------------------------------------
Acquiring ready buffers to process
-------------------------------------------------------------------------------
Ready buffers can be acquired using v4l2_m2m_next_src_buf()/v4l2_m2m_next_dst_buf(). After the transaction they are removed from the queues with v4l2_m2m_dst_buf_remove()/v4l2_m2m_src_buf_remove(). This is not multi-buffer-transaction-safe. It will have to be modified, but ideally after we decide how to handle multi-buffer transactions in videobuf core.-------------------------------------------------------------------------------
poll()
-------------------------------------------------------------------------------
We cannot havepoll()for multiple queues on one node, so we usepoll()for the destination queue only.-------------------------------------------------------------------------------
mmap()
-------------------------------------------------------------------------------
Requirements:
- allow mapping buffers from different queues
- retain "magic" offset values so videobuf can still match buffers by offsetsThe proposed solution involves a
querybuf()andmmap()multiplexers:a) When a driver calls
querybuf(), we have access to the type and we can detect which queue to callvideobuf_querybuf()on:vq = v4l2_m2m_get_vq(m2m_ctx, buf->type); ret = videobuf_querybuf(vq, buf);The offsets returned from
videobuf_querybuf()for one of the queues are further offset by a predefined constant (DST_QUEUE_OFF_BASE). This way the driver (and applications) receive different offsets for the same buffer indexes of each queue:if (buf->memory == V4L2_MEMORY_MMAP && vq->type == V4L2_BUF_TYPE_VIDEO_CAPTURE) { buf->m.offset += DST_QUEUE_OFF_BASE; }b) When the application (driver) calls
mmap(), the offsets which were modified inquerybuf()are detected and the proper queue for them chosen based on that. Finally, the modified offsets are passed tovideobuf_mmap_mapper()for proper queues with their offsets changed back to values recognizable by videobuf:unsigned long offset = vma->vm_pgoff << PAGE_SHIFT; struct videobuf_queue *vq; if (offset < DST_QUEUE_OFF_BASE) { vq = v4l2_m2m_get_src_vq(m2m_ctx); } else { vq = v4l2_m2m_get_dst_vq(m2m_ctx); vma->vm_pgoff -= (DST_QUEUE_OFF_BASE >> PAGE_SHIFT); } return videobuf_mmap_mapper(vq, vma);-------------------------------------------------------------------------------
Test device and a userspace application
-------------------------------------------------------------------------------mem2mem_testdev.cis a test driver for the framework. It uses timers for fake interrupts and allows testing transaction with different number of buffers and transaction durations simultaneously.
process-vmalloc.cis a capture+output test application for the test device.-------------------------------------------------------------------------------
Future work
-------------------------------------------------------------------------------
- read/write support
- transaction/abort timeouts
- extracting more common code to the framework? (e.g. per-queue format details, transaction length, etc.)Best regards
--
Pawel Osciak
Linux Platform Group
Samsung Poland R&D Center--
To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majordomo at vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 1 line
Hans Verkuil - video4linux developer - sponsored by TANDBERG
cover Mem-to-mem device framework
Hello Hans,
Quoted 5 lines · On Wednesday 23 December 2009 16:06:18 Hans Verkuil wrote:
Thank you for working on this! It's much appreciated. Now I've noticed that patches regarding memory-to-memory and memory pool tend to get very few comments. I suspect that the main reason is that these are SoC-specific features that do not occur in consumer-type products. So most v4l developers do not have the interest and motivation (and time!) to look into this.
Thank you very much for your response. We were a bit surprised with the lack of responses as there seemed to be a good number of people interested in this area.
I'm hoping that everybody interested would take a look at the test device posted along with the patches. It's virtual, no specific hardware required, but it demonstrates the concepts behind the framework, including transactions.
Quoted 8 lines
One thing that I am missing is a high-level overview of what we want. Currently there are patches/RFCs floating around for memory-to-memory support, multiplanar support and memory-pool support.
What I would like to see is a RFC that ties this all together from the point of view of the public API. I.e. what are the requirements? Possibly solutions? Open questions? Forget about how to implement it for the moment, that will follow from the chosen solutions.
Yes, that's true, sorry about that. We've been so into it after the memory pool discussion and the V4L2 mini-summit that I neglected describing the big picture behind this.
So to give a more high-level description, from the point of view of applications and the V4L2 API:
---------------
Requirements:
---------------
(Some of the following were first posted by Laurent in:
http://thread.gmane.org/gmane.linux.drivers.video-input-infrastructure/10204).
1. Support for devices that take input data in a source buffer, take a separate destination buffer, process the source data and put it in the destination buffer.
2. Allow sharing buffers between devices, effectively chaining them to form video pipelines. An example of this could be a video decoder, fed with video stream which returns raw frames, which then have to be postprocessed by another device and displayed. This is the main scenario we need to have for our S3C/S5P series SoCs. Of course, we'd like zero-copy.
3. Allow using more than one buffer by the device at the same time. This is not supported by videobuffer (e.g. we have to choose on which buffer we'd like to sleep, and we do not always know that). This is not really a requirement from the V4L2 API point of view, but has direct influence on how poll() and blocking I/O works.
4. Multiplanar buffers. Our devices require them (see the RFC for more details:
http://article.gmane.org/gmane.linux.drivers.video-input-infrastructure/11212).
5. Solve problems with cache coherency on non-x86 architectures, especially in videobuf for OUTPUT buffers. We need to flush the cache before starting the transaction.
6. Reduce buffer queuing latency, e.g.: move operations such as locking, out of qbuf.
Applications would like to queue a buffer and be able to fire up the device as fast as possible.
7. Large buffer allocations, buffer preallocation, etc.
---------------
Solutions:
---------------
1. After a detailed discussion, we agreed in:
http://thread.gmane.org/gmane.linux.drivers.video-input-infrastructure/10668,
that we'd like the application to be able to queue/dequeue both OUTPUT (as source) and CAPTURE (as destination) buffers on one video node. Activating the device (after streamon) would take effect only if there are both types of buffers available. The application would put source data into OUTPUT buffers and expect to find it processed in dequeued CAPTURE buffers. Addressed by mem2mem framework.
2. I don't see anything to do here from the API's point of view. The application would open two video nodes, e.g. video decoder and video postprocessor and queue buffers dequeued from decoder on the postprocessor. To get the best performance, this requires the buffers to be marked as non cached somehow to avoid unneeded cache syncs.
3. Mem2mem addresses this partially by adding a "transaction" concept. It's not bullet-proof though, as it assumes the buffers will be returned in the same order as passed. Some videobuffer limitations will have to be addressed here.
4. See my RFC. Patches in progress.
5. We have narrowed it down to an additional sync() before the operation (i.e. in qbuf), but more issues may exist here. I have already added sync() support for qbuf with minimal changes to videobuf and will be posting the proposal soon. This also requires identifying the direction of the sync, but we have found a way to do this without adding anything new (videobuf flags are enough).
6. Later. We haven't done anything in this field.
7. We use our own allocator (see
http://thread.gmane.org/gmane.linux.ports.arm.kernel/56879), but we have a new concept for that which we'd like to discuss separately later.
Quoted 6 lines
Note that I would suggest though that the memory-pool part is split into two
parts: how to actually allocate the memory is pretty much separate from how v4l will use it. The actual allocation part is probably quite complex and might even be hardware dependent and should be discussed separately. But how to use it is something that can be discussed without needing to know how it was allocated.
Exactly, this is the approach we have assumed right now. We'd like to introduce each part of the infrastructure incrementally. The plan was to do exactly as you said: leave the allocator-specific parts for later and for a separate discussion.
We intend to follow with multi-planar buffers and then to focus on dma-contig, as this is what our hardware requires.
Quoted 6 lines
BTW, what is the status of the multiplanar RFC? I later realized that that RFC might be very useful for adding meta-data to buffers. There are several cases where that is useful: sensors that provide meta-data when capturing a frame and imagepipelines (particularly in memory-to-memory cases) that want to have all parameters as part of the meta-data associated with the image. There may well be more of those.
This got pushed back but now after m2m, it's become next task on my list. I expect to be posting patches in a week or two, hopefully.
I understand that you'd like to make the pointer in the union and the helper struct more generic to use it to pass different types of information?
Best regards
--
Pawel Osciak
Linux Platform Group
Samsung Poland R&D Center
cover Mem-to-mem device framework
Quoted 30 lines
-----Original Message-----
From: Pawel Osciak [mailto:p.osciak at samsung.com]
Sent: Monday, December 28, 2009 8:19 PM
To: 'Hans Verkuil'
Cc: linux-media at vger.kernel.org; linux-samsung-soc at vger.kernel.org; linux-arm-kernel at lists.infradead.org; Marek Szyprowski; kyungmin.park at samsung.com; Hiremath, Vaibhav; Karicheri, Muralidharan; 'Guru Raj'; 'Xiaolin Zhang'; 'Magnus Damm'; 'Sakari Ailus'
Subject: RE: [PATCH/RFC v2.1 0/2] Mem-to-mem device frameworkHello Hans,
Quoted 1 line · On Wednesday 23 December 2009 16:06:18 Hans Verkuil wrote:
Thank you for working on this! It's much appreciated. Now I've
noticed that
Quoted 1 line
patches regarding memory-to-memory and memory pool tend to get
very few comments.
Quoted 1 line
I suspect that the main reason is that these are SoC-specific
features that do
Quoted 1 line
not occur in consumer-type products. So most v4l developers do not
have the
Quoted 1 line
interest and motivation (and time!) to look into this.
Thank you very much for your response. We were a bit surprised with the lack of
responses as there seemed to be a good number of people interested in this area.
Quoted 7 lines
I'm hoping that everybody interested would take a look at the test device posted
along with the patches. It's virtual, no specific hardware required, but it
demonstrates the concepts behind the framework, including transactions.
[Hiremath, Vaibhav] I was on vacation and resumed today itself, I will go through these patch series this weekend and will get back to you.
I just had cursory look and I would say it should be really good starting point for us to support mem-to-mem devices.
Thanks,
Vaibhav
Quoted 185 lines
Quoted 1 line
One thing that I am missing is a high-level overview of what we
want. Currently
Quoted 1 line
there are patches/RFCs floating around for memory-to-memory
support, multiplanar
Quoted 3 lines
support and memory-pool support.
What I would like to see is a RFC that ties this all together from
the point of
Quoted 1 line
view of the public API. I.e. what are the requirements? Possibly
solutions? Open
Quoted 1 line
questions? Forget about how to implement it for the moment, that
will follow
Quoted 1 line
from the chosen solutions.
Yes, that's true, sorry about that. We've been so into it after the memory pool
discussion and the V4L2 mini-summit that I neglected describing the big picture
behind this.So to give a more high-level description, from the point of view of applications
and the V4L2 API:---------------
Requirements:
---------------
(Some of the following were first posted by Laurent in:
http://thread.gmane.org/gmane.linux.drivers.video-input-
infrastructure/10204).1. Support for devices that take input data in a source buffer, take a separate
destination buffer, process the source data and put it in the destination buffer.2. Allow sharing buffers between devices, effectively chaining them to form
video pipelines. An example of this could be a video decoder, fed with video
stream which returns raw frames, which then have to be postprocessed by another
device and displayed. This is the main scenario we need to have for our S3C/S5P
series SoCs. Of course, we'd like zero-copy.3. Allow using more than one buffer by the device at the same time. This is not
supported by videobuffer (e.g. we have to choose on which buffer we'd like
to sleep, and we do not always know that). This is not really a requirement
from the V4L2 API point of view, but has direct influence on howpoll()and
blocking I/O works.4. Multiplanar buffers. Our devices require them (see the RFC for more details:
http://article.gmane.org/gmane.linux.drivers.video-input-
infrastructure/11212).5. Solve problems with cache coherency on non-x86 architectures, especially in
videobuf for OUTPUT buffers. We need to flush the cache before starting the
transaction.6. Reduce buffer queuing latency, e.g.: move operations such as locking, out
of qbuf.
Applications would like to queue a buffer and be able to fire up the device
as fast as possible.7. Large buffer allocations, buffer preallocation, etc.
---------------
Solutions:
---------------
1. After a detailed discussion, we agreed in:
http://thread.gmane.org/gmane.linux.drivers.video-input-
infrastructure/10668,
that we'd like the application to be able to queue/dequeue both OUTPUT (as source)
and CAPTURE (as destination) buffers on one video node. Activating the device
(after streamon) would take effect only if there are both types of buffers
available. The application would put source data into OUTPUT buffers and expect
to find it processed in dequeued CAPTURE buffers. Addressed by mem2mem framework.2. I don't see anything to do here from the API's point of view. The application
would open two video nodes, e.g. video decoder and video postprocessor and queue
buffers dequeued from decoder on the postprocessor. To get the best performance,
this requires the buffers to be marked as non cached somehow to avoid unneeded
cache syncs.3. Mem2mem addresses this partially by adding a "transaction" concept. It's
not bullet-proof though, as it assumes the buffers will be returned in the same
order as passed. Some videobuffer limitations will have to be addressed here.4. See my RFC. Patches in progress.
5. We have narrowed it down to an additional
sync()before the operation
(i.e. in qbuf), but more issues may exist here. I have already addedsync()
support for qbuf with minimal changes to videobuf and will be posting the
proposal soon. This also requires identifying the direction of the sync, but
we have found a way to do this without adding anything new (videobuf flags
are enough).6. Later. We haven't done anything in this field.
7. We use our own allocator (see
http://thread.gmane.org/gmane.linux.ports.arm.kernel/56879), but we have a new
concept for that which we'd like to discuss separately later.Quoted 1 line
Note that I would suggest though that the memory-pool part is
split into two
Quoted 1 line
parts: how to actually allocate the memory is pretty much separate
from how v4l
Quoted 1 line
will use it. The actual allocation part is probably quite complex
and might
Quoted 1 line
even be hardware dependent and should be discussed separately. But
how to use
Quoted 1 line
it is something that can be discussed without needing to know how
it was
Quoted 1 line
allocated.
Exactly, this is the approach we have assumed right now. We'd like to introduce
each part of the infrastructure incrementally. The plan was to do exactly as
you said: leave the allocator-specific parts for later and for a separate
discussion.
We intend to follow with multi-planar buffers and then to focus on dma-contig,
as this is what our hardware requires.Quoted 1 line
BTW, what is the status of the multiplanar RFC? I later realized
that that RFC
Quoted 1 line
might be very useful for adding meta-data to buffers. There are
several cases
Quoted 1 line
where that is useful: sensors that provide meta-data when
capturing a frame and
Quoted 1 line
imagepipelines (particularly in memory-to-memory cases) that want
to have all
Quoted 1 line
parameters as part of the meta-data associated with the image.
There may well
Quoted 1 line
be more of those.
This got pushed back but now after m2m, it's become next task on my list. I
expect to be posting patches in a week or two, hopefully. I understand that you'd like to make the pointer in the union and the helper
struct more generic to use it to pass different types of information?Best regards
--
Pawel Osciak
Linux Platform Group
Samsung Poland R&D Center
cover Mem-to-mem device framework
Quoted 36 lines · Vaibhav
Quoted 29 lines
-----Original Message-----
From: Pawel Osciak [mailto:p.osciak at samsung.com]
Sent: Monday, December 28, 2009 8:19 PM
To: 'Hans Verkuil'
Cc: linux-media at vger.kernel.org; linux-samsung-soc at vger.kernel.org; linux-arm-kernel at lists.infradead.org; Marek Szyprowski; kyungmin.park at samsung.com; Hiremath, Vaibhav; Karicheri, Muralidharan; 'Guru Raj'; 'Xiaolin Zhang'; 'Magnus Damm'; 'Sakari Ailus'
Subject: RE: [PATCH/RFC v2.1 0/2] Mem-to-mem device frameworkHello Hans,
Quoted 1 line · On Wednesday 23 December 2009 16:06:18 Hans Verkuil wrote:
Thank you for working on this! It's much appreciated. Now I've
noticed that
Quoted 1 line
patches regarding memory-to-memory and memory pool tend to get
very few comments.
Quoted 1 line
I suspect that the main reason is that these are SoC-specific
features that do
Quoted 1 line
not occur in consumer-type products. So most v4l developers do not
have the
Quoted 1 line
interest and motivation (and time!) to look into this.
Thank you very much for your response. We were a bit surprised with the lack of
responses as there seemed to be a good number of people interested in this area.Quoted 7 lines
I'm hoping that everybody interested would take a look at the test device posted
along with the patches. It's virtual, no specific hardware required, but it
demonstrates the concepts behind the framework, including transactions.
Quoted 3 lines
[Hiremath, Vaibhav] I was on vacation and resumed today itself, I will go through these patch series this weekend and will get back to you.
I just had cursory look and I would say it should be really good starting point for us to support mem-to-mem devices.\
Hmm... it seems to me that those patches are still under discussion/analysis. I'll mark them as RFC at the Patchwork.
Please let me know after you, SoC guys, go into a consensus about it. Then, please submit me the final version.
Cheers,
Mauro