frame-size switching: preview / single-shot use-case
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines · Mauro Carvalho Chehab
Quoted 4 lines
There's an additional problem with that: assume that streaming is happening, and a
S_FMTchanging the resolution was sent. There's no way to warrant that the very next frame will have the new
resolution. So, a meta-data with the frame resolution (and format)
would be needed.Em 17-02-2011 17:26, Guennadi Liakhovetski escreveu:
Quoted 3 lines
Sorry, we are not going to allow format changes during a running capture. You have to stop streaming, set new formats (possibly switch to another queue) and restart streaming.
Quoted 6 lines
Quoted 1 line
Quoted 1 line
What am I missing?
Quoted 3 lines · Mauro Carvalho Chehab
If you're stopping the stream, the current API will work as-is.
If all of your concerns is about reserving a bigger buffer queue, I think that one of the reasons for the CMA allocator it for such usage.
From: Guennadi Liakhovetski <[email protected]>
Quoted 2 lines
Not just bigger, say, with our preview / still-shot example, we would have one queue with a larger number of small buffers for drop-free preview, and a small number of larger buffers for still images.
Ie. waste memory? As in you have both those queues allocated but only one is used at given time?
Quoted 7 lines
Currently you would have to allocate a large number of large buffers, which would waste memory. Or you would have to reallocate the queue, losing time.
AFAICS, CMA doesn't manage our memory for us. It only provides an API to reserve memory for various uses with various restrictions (alignment, etc.) and use different allocators to obtain that memory.
I'm not sure if I understand you here. CMA has some API for reserving memory at boot time but it sure does manage this reserved memory, ie. when system is running you can allocate chunks of memory from this reserved block.
Also note, that latest CMA uses only one allocator.
Quoted 5 lines
So, are you suggesting, that with that in place, we would first allocate the preview queue from this memory, then free it, when switching to snapshooting, allocate our large-buffer queue from the _same_ memory, capture images, free and allocate preview queue again? Would that be fast enough?
If CMA is considered, the most important thing to note is that CMA may share memory with page allocator (so that other parts of the system can use it if CMA-compatible devices are not using it). When CMA allocates memory chunk it may potentially need to migrate memory pages which may take so time (there is room for improvement, but still).
Sharing can be disabled in which case allocation should be quite fast (the last CMA patchset uses a first-fit bitmap-based gen_allocator API but O(log n) best-fit algorithm can easily used instead).
To sum things up, if sharing is disabled, CMA should be able to fulfil your requirements, however it may be undesirable as it wastes space. If sharing is enabled, on the other hand, the delay may potentially be noticeable.
Quoted 2 lines
In fact, it would be kind of smart to reuse the same memory for both queues, but if we could do it without re-allocations?...
What I would do is allocate a few big buffers and when needed divide them into smaller chunks (or even allocate one big block and later divide it in whatever way needed). I'm not sure if such usage would map well to V4L2 API.
This usage is, as a matter of fact, supported by CMA. You can allocate a big block and then run cma_create() on it to create a new CMA context. Using this context you can allocate a lot of small blocks, then free them all, to finally allocate few big blocks.
Again, I'm not sure how it maps to V4L2 API. If you can change formats while retaining V4L device instance's state, this should be doable.
Signature · 4 lines
Best regards, _ _ .o. | Liege of Serenely Enlightened Majesty of o' \,=./ `o ..o | Computer Science, Michal "mina86" Nazarewicz (o o) ooo +-----<email/xmpp: [email protected]>-----ooO--(_)--Ooo--
Hi Michal,
Quoted 20 lines · On Friday 18 February 2011 00:09:51 Michal Nazarewicz wrote:
Quoted 12 lines
Quoted 12 lines
Quoted 6 lines · Mauro Carvalho Chehab
Quoted 5 lines
There's an additional problem with that: assume that streaming is happening, and a
S_FMTchanging the resolution was sent. There's no way to warrant that the very next frame will have the new resolution. So, a meta-data with the frame resolution (and format) would be needed.Em 17-02-2011 17:26, Guennadi Liakhovetski escreveu:
Quoted 4 lines
Sorry, we are not going to allow format changes during a running capture. You have to stop streaming, set new formats (possibly switch to another queue) and restart streaming.
Quoted 7 lines
Quoted 1 line
Quoted 1 line
What am I missing?
Quoted 4 lines · Mauro Carvalho Chehab
If you're stopping the stream, the current API will work as-is.
If all of your concerns is about reserving a bigger buffer queue, I think that one of the reasons for the CMA allocator it for such usage.
Quoted 8 lines
From: Guennadi Liakhovetski <[email protected]>
Quoted 3 lines
Not just bigger, say, with our preview / still-shot example, we would have one queue with a larger number of small buffers for drop-free preview, and a small number of larger buffers for still images.
Ie. waste memory? As in you have both those queues allocated but only one is used at given time?
It's a trade-off between memory and speed. Preallocating still image capture buffers will give you better snapshot performances, at the expense of memory.
The basic problems we have here is that taking snapshots is slow with the current API if we need to stop capture, free buffers, change the format, allocate new buffers (and perform cache management operations) and restart the stream. To fix this we're considering a way to preallocate still image capture buffers, but I'm open to proposals for other ways to solve the issue :-)
Quoted 51 lines
Quoted 7 lines
Currently you would have to allocate a large number of large buffers, which would waste memory. Or you would have to reallocate the queue, losing time.
AFAICS, CMA doesn't manage our memory for us. It only provides an API to reserve memory for various uses with various restrictions (alignment, etc.) and use different allocators to obtain that memory.
I'm not sure if I understand you here. CMA has some API for reserving memory at boot time but it sure does manage this reserved memory, ie. when system is running you can allocate chunks of memory from this reserved block.
Also note, that latest CMA uses only one allocator.
Quoted 5 lines
So, are you suggesting, that with that in place, we would first allocate the preview queue from this memory, then free it, when switching to snapshooting, allocate our large-buffer queue from the _same_ memory, capture images, free and allocate preview queue again? Would that be fast enough?
If CMA is considered, the most important thing to note is that CMA may share memory with page allocator (so that other parts of the system can use it if CMA-compatible devices are not using it). When CMA allocates memory chunk it may potentially need to migrate memory pages which may take so time (there is room for improvement, but still).
Sharing can be disabled in which case allocation should be quite fast (the last CMA patchset uses a first-fit bitmap-based
gen_allocatorAPI but O(log n) best-fit algorithm can easily used instead).To sum things up, if sharing is disabled, CMA should be able to fulfil your requirements, however it may be undesirable as it wastes space. If sharing is enabled, on the other hand, the delay may potentially be noticeable.
Quoted 2 lines
In fact, it would be kind of smart to reuse the same memory for both queues, but if we could do it without re-allocations?...
What I would do is allocate a few big buffers and when needed divide them into smaller chunks (or even allocate one big block and later divide it in whatever way needed). I'm not sure if such usage would map well to V4L2 API.
This usage is, as a matter of fact, supported by CMA. You can allocate a big block and then run
cma_create()on it to create a new CMA context. Using this context you can allocate a lot of small blocks, then free them all, to finally allocate few big blocks.Again, I'm not sure how it maps to V4L2 API. If you can change formats while retaining V4L device instance's state, this should be doable.
Signature · 3 lines
Regards, Laurent Pinchart
Quoted 33 lines · Laurent Pinchart
Hi Michal,
Quoted 21 lines · On Friday 18 February 2011 00:09:51 Michal Nazarewicz wrote:
Quoted 12 lines
Quoted 12 lines
Quoted 6 lines · Mauro Carvalho Chehab
Quoted 5 lines
There's an additional problem with that: assume that streaming is happening, and a
S_FMTchanging the resolution was sent. There's no way to warrant that the very next frame will have the new resolution. So, a meta-data with the frame resolution (and format) would be needed.Em 17-02-2011 17:26, Guennadi Liakhovetski escreveu:
Quoted 4 lines
Sorry, we are not going to allow format changes during a running capture. You have to stop streaming, set new formats (possibly switch to another queue) and restart streaming.
Quoted 7 lines
Quoted 1 line
Quoted 1 line
What am I missing?
Quoted 4 lines · Mauro Carvalho Chehab
If you're stopping the stream, the current API will work as-is.
If all of your concerns is about reserving a bigger buffer queue, I think that one of the reasons for the CMA allocator it for such
usage.
Quoted 8 lines
From: Guennadi Liakhovetski <[email protected]>
Quoted 3 lines
Not just bigger, say, with our preview / still-shot example, we would have one queue with a larger number of small buffers for drop-free preview, and a small number of larger buffers for still images.
Ie. waste memory? As in you have both those queues allocated but only one is used at given time?
Quoted 8 lines
It's a trade-off between memory and speed. Preallocating still image capture buffers will give you better snapshot performances, at the expense of memory.
The basic problems we have here is that taking snapshots is slow with the current API if we need to stop capture, free buffers, change the format, allocate new buffers (and perform cache management operations) and restart the stream. To fix this we're considering a way to preallocate still image capture buffers, but I'm open to proposals for other ways to solve the issue :-)
What I'm trying to say is that it would be best if one could configure the device in such a way that switching between modes would not require the device to free buffers (even though in user space they could be inaccessible).
This is what I have in mind the usage would look like:
1. Open device
Kernel creates some control structures, the usual stuff.
2. Initialize multi-format (specifying what formats user space will use). Kernel calculates amount of memory needed for the most demanding format and allocates it.
3. Set format (restricted to one of formats specified in step 2) Kernel has memory already allocated, so it only needs to split it to buffers needed in given format.
4. Allocate buffers.
Kernel allocates memory needed for most demanding format (calculated in step 2).
Once memory is allocated, splits it to buffers needed in given format.
5. Do the loop... queue, dequeue, all the usual stuff.
Kernel instructs device to handle buffers, the usual stuff.
6. Free buffers.
Kernel space destroys the buffers needed for given format but DOES NOT free memory.
7. If not done, go to step 3.
8. Finish multi-format mode.
Kernel actually frees the memory.
9. Close the device.
A V4L device driver could just ignore step 2 and 7 and work in the less optimal mode.
If I understand V4L2 correctly, the API does not allow for step 2 and 8. In theory, they could be merged with step 1 and 9 respectively, I don't know id that feasible though.
Signature · 4 lines
Best regards, _ _ .o. | Liege of Serenely Enlightened Majesty of o' \,=./ `o ..o | Computer Science, Michal "mina86" Nazarewicz (o o) ooo +-----<email/xmpp: [email protected]>-----ooO--(_)--Ooo--
Em 18-02-2011 08:31, Laurent Pinchart escreveu:
Quoted 10 lines
It's a trade-off between memory and speed. Preallocating still image capture buffers will give you better snapshot performances, at the expense of memory.
The basic problems we have here is that taking snapshots is slow with the current API if we need to stop capture, free buffers, change the format, allocate new buffers (and perform cache management operations) and restart the stream. To fix this we're considering a way to preallocate still image capture buffers, but I'm open to proposals for other ways to solve the issue :-)
From the above operations, considering that CMA is used to reserve a
non-shared memory with enough space for the new buffer size/qtd, I don't think that the most expensive operation would be to realloc the memory.
The logic to stop/start streaming seems to be the most consuming one, as driver will need to wait for the current I/O operation to complete, and this can take hundreds of milisseconds (the duration of one frame).
How much time would CMA need to free and re-allocate the buffers for, let's say, something in the range of 1-10 MB, on a pre-allocated, non shared memory space?
Cheers
Mauro
Hi Mauro,
Quoted 16 lines · On Friday 18 February 2011 13:33:12 Mauro Carvalho Chehab wrote:
Em 18-02-2011 08:31, Laurent Pinchart escreveu:
Quoted 10 lines
It's a trade-off between memory and speed. Preallocating still image capture buffers will give you better snapshot performances, at the expense of memory.
The basic problems we have here is that taking snapshots is slow with the current API if we need to stop capture, free buffers, change the format, allocate new buffers (and perform cache management operations) and restart the stream. To fix this we're considering a way to preallocate still image capture buffers, but I'm open to proposals for other ways to solve the issue :-)
From the above operations, considering that CMA is used to reserve a non-shared memory with enough space for the new buffer size/qtd, I don't think that the most expensive operation would be to realloc the memory.
Quoted 7 lines
The logic to stop/start streaming seems to be the most consuming one, as driver will need to wait for the current I/O operation to complete, and this can take hundreds of milisseconds (the duration of one frame).
How much time would CMA need to free and re-allocate the buffers for, let's say, something in the range of 1-10 MB, on a pre-allocated, non shared memory space?
CMA won't solve the problem, as not all drivers require continuous memory. The OMAP3 ISP has an IOMMU, and two very time-consuming tasks are mapping the memory to the IOMMU and cleaning the cache. This is why preallocation *and* prequeuing is needed.
Signature · 3 lines
Regards, Laurent Pinchart
Quoted 9 lines
Em 18-02-2011 08:31, Laurent Pinchart escreveu:
Quoted 8 lines
It's a trade-off between memory and speed. Preallocating still image capture buffers will give you better snapshot performances, at the expense of memory.
The basic problems we have here is that taking snapshots is slow with the current API if we need to stop capture, free buffers, change the format, allocate new buffers (and perform cache management operations)
and restart the stream. To fix this we're considering a way to
preallocate still image capture buffers, but I'm open to proposals for
other ways to solve the issue :-)
Quoted 9 lines · Mauro Carvalho Chehab
From the above operations, considering that CMA is used to reserve a non-shared memory with enough space for the new buffer size/qtd, I don't think that the most expensive operation would be to realloc the memory.
The logic to stop/start streaming seems to be the most consuming one, as driver will need to wait for the current I/O operation to complete, and this can take hundreds of milisseconds (the duration of one frame).
How much time would CMA need to free and re-allocate the buffers for, let's say, something in the range of 1-10 MB, on a pre-allocated, non shared memory space?
Like I said, if memory is shared with page allocator this can potentially take a lot of time. If device driver could preallocate some space and then spit it into buffers when needed, it could be a better solution then to free chunk and allocate a new one. CMA was not designed with very low latency in mind.
Signature · 4 lines
Best regards, _ _ .o. | Liege of Serenely Enlightened Majesty of o' \,=./ `o ..o | Computer Science, Michal "mina86" Nazarewicz (o o) ooo +-----<email/xmpp: [email protected]>-----ooO--(_)--Ooo--
Hi Michal,
On Friday 18 February 2011 12:37:24 Michal Nazarewicz wrote:
[snip]
Quoted 6 lines
What I'm trying to say is that it would be best if one could configure the device in such a way that switching between modes would not require the device to free buffers (even though in user space they could be inaccessible).
Quoted 7 lines
This is what I have in mind the usage would look like:
1. Open device
Kernel creates some control structures, the usual stuff.
2. Initialize multi-format (specifying what formats user space will use). Kernel calculates amount of memory needed for the most demanding format and allocates it.
Don't forget that applications can also use USERPTR. We need a low-latency solution for that as well.
Quoted 10 lines
3. Set format (restricted to one of formats specified in step 2) Kernel has memory already allocated, so it only needs to split it to buffers needed in given format.
4. Allocate buffers.
Kernel allocates memory needed for most demanding format (calculated in step 2).
Once memory is allocated, splits it to buffers needed in given format.
5. Do the loop... queue, dequeue, all the usual stuff.
Kernel instructs device to handle buffers, the usual stuff.
When buffers are queued cache needs to be cleaned. This is an expensive operation, and we need to be able to pre-queue (or at least pre-clean) buffers.
Quoted 17 lines
6. Free buffers.
Kernel space destroys the buffers needed for given format but DOES NOT free memory.7. If not done, go to step 3.
8. Finish multi-format mode.
Kernel actually frees the memory.9. Close the device.
A V4L device driver could just ignore step 2 and 7 and work in the less optimal mode.
If I understand V4L2 correctly, the API does not allow for step 2 and 8. In theory, they could be merged with step 1 and 9 respectively, I don't know id that feasible though.
Signature · 3 lines
Regards, Laurent Pinchart
Quoted 13 lines · Laurent Pinchart
Hi Michal,
On Friday 18 February 2011 12:37:24 Michal Nazarewicz wrote:
[snip]
Quoted 7 lines
What I'm trying to say is that it would be best if one could configure the device in such a way that switching between modes would not require the device to free buffers (even though in user space they could be
inaccessible).This is what I have in mind the usage would look like:
Quoted 7 lines
Quoted 5 lines
1. Open device
Kernel creates some control structures, the usual stuff.
2. Initialize multi-format (specifying what formats user space will use). Kernel calculates amount of memory needed for the most demanding format and allocates it.Don't forget that applications can also use USERPTR. We need a low-latency solution for that as well.
That would probably work best if user provided one big buffer. Again, I don't know how this maps to V4L API.
Quoted 12 lines
Quoted 11 lines
3. Set format (restricted to one of formats specified in step 2) Kernel has memory already allocated, so it only needs to split it to buffers needed in given format.
4. Allocate buffers.
Kernel allocates memory needed for most demanding format (calculated in step 2).
Once memory is allocated, splits it to buffers needed in given format.
5. Do the loop... queue, dequeue, all the usual stuff.
Kernel instructs device to handle buffers, the usual stuff.
Quoted 3 lines
When buffers are queued cache needs to be cleaned. This is an expensive operation, and we need to be able to pre-queue (or at least pre-clean) buffers.
Cache operations are always needed, aren't they? Whatever you do, you will always have to handle cache coherency (in one way or another) so there's nothing we can do about it, or is there?
Quoted 17 lines
Quoted 17 lines
6. Free buffers.
Kernel space destroys the buffers needed for given format but DOES NOT free memory.7. If not done, go to step 3.
8. Finish multi-format mode.
Kernel actually frees the memory.9. Close the device.
A V4L device driver could just ignore step 2 and 7 and work in the less optimal mode.
If I understand V4L2 correctly, the API does not allow for step 2 and 8. In theory, they could be merged with step 1 and 9 respectively, I don't know id that feasible though.
Signature · 4 lines
Best regards, _ _ .o. | Liege of Serenely Enlightened Majesty of o' \,=./ `o ..o | Computer Science, Michal "mina86" Nazarewicz (o o) ooo +-----<email/xmpp: [email protected]>-----ooO--(_)--Ooo--
Hi Michal,
Quoted 45 lines · On Friday 18 February 2011 14:19:44 Michal Nazarewicz wrote:
Quoted 16 lines · Laurent Pinchart
On Friday 18 February 2011 12:37:24 Michal Nazarewicz wrote:
[snip]
Quoted 12 lines
What I'm trying to say is that it would be best if one could configure the device in such a way that switching between modes would not require the device to free buffers (even though in user space they could be inaccessible).
This is what I have in mind the usage would look like:
1. Open device
Kernel creates some control structures, the usual stuff.
Quoted 8 lines
Quoted 5 lines
2. Initialize multi-format (specifying what formats user space will use).
Kernel calculates amount of memory needed for the most demanding format and allocates it.
Don't forget that applications can also use USERPTR. We need a low-latency solution for that as well.
That would probably work best if user provided one big buffer. Again, I don't know how this maps to V4L API.
Quoted 16 lines
Quoted 15 lines
3. Set format (restricted to one of formats specified in step 2)
Kernel has memory already allocated, so it only needs to split it to buffers needed in given format.
4. Allocate buffers.
Kernel allocates memory needed for most demanding format (calculated in step 2).
Once memory is allocated, splits it to buffers needed in given format.5. Do the loop... queue, dequeue, all the usual stuff.
Kernel instructs device to handle buffers, the usual stuff.
Quoted 7 lines
Quoted 3 lines
When buffers are queued cache needs to be cleaned. This is an expensive operation, and we need to be able to pre-queue (or at least pre-clean) buffers.
Cache operations are always needed, aren't they? Whatever you do, you will always have to handle cache coherency (in one way or another) so there's nothing we can do about it, or is there?
To achieve low latency still image capture, you need to minimize the time spent reconfiguring the device from viewfinder to still capture. Cache cleaning is always needed, but you can prequeue buffers you can clean the cache in advance, avoiding an extra delay when the user presses the still image capture button.
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
6. Free buffers.
Kernel space destroys the buffers needed for given format but DOES NOT free memory.
7. If not done, go to step 3.
8. Finish multi-format mode.
Kernel actually frees the memory.
9. Close the device.
A V4L device driver could just ignore step 2 and 7 and work in the less optimal mode.
If I understand V4L2 correctly, the API does not allow for step 2 and 8. In theory, they could be merged with step 1 and 9 respectively, I don't know id that feasible though.
Signature · 3 lines
Regards, Laurent Pinchart
Quoted 4 lines · On Friday 18 February 2011 14:19:44 Michal Nazarewicz wrote:
Quoted 3 lines
Cache operations are always needed, aren't they? Whatever you do, you will always have to handle cache coherency (in one way or another) so there's nothing we can do about it, or is there?
Quoted 5 lines · Laurent Pinchart
To achieve low latency still image capture, you need to minimize the time spent reconfiguring the device from viewfinder to still capture. Cache cleaning is always needed, but you can prequeue buffers you can clean the cache in advance, avoiding an extra delay when the user presses the still image capture button.
If there is enough time to perform those operation while preview is shown (ie. several frames pare second), why would there not be enough time to perform those operations for still image capture?
If I understand you correctly, what you are describing is a situation where one has set of buffers for preview and a buffer for still image laying around waiting to be used, right?
Such scheme will of course work, but I'm just suggesting to think of a scheme where the unused buffer for still image is reused for preview frames when preview is shown.
Signature · 4 lines
Best regards, _ _ .o. | Liege of Serenely Enlightened Majesty of o' \,=./ `o ..o | Computer Science, Michal "mina86" Nazarewicz (o o) ooo +-----<email/xmpp: [email protected]>-----ooO--(_)--Ooo--
Hi Michal,
Quoted 12 lines · On Friday 18 February 2011 15:05:42 Michal Nazarewicz wrote:
Quoted 4 lines · On Friday 18 February 2011 14:19:44 Michal Nazarewicz wrote:
Quoted 3 lines
Cache operations are always needed, aren't they? Whatever you do, you will always have to handle cache coherency (in one way or another) so there's nothing we can do about it, or is there?
Quoted 5 lines · Laurent Pinchart
To achieve low latency still image capture, you need to minimize the time spent reconfiguring the device from viewfinder to still capture. Cache cleaning is always needed, but you can prequeue buffers you can clean the cache in advance, avoiding an extra delay when the user presses the still image capture button.
Quoted 3 lines
If there is enough time to perform those operation while preview is shown (ie. several frames pare second), why would there not be enough time to perform those operations for still image capture?
For still image capture you want to minimize the shot-to-snapshot delay (delay between the time when the user presses the shot button and the time when the image is captured). Allocating memory for the buffers and prequeuing them should thus be done before the user presses the shot button.
Quoted 3 lines
If I understand you correctly, what you are describing is a situation where one has set of buffers for preview and a buffer for still image laying around waiting to be used, right?
That's correct. That's what we currently do with the OMAP3 ISP.
Quoted 3 lines
Such scheme will of course work, but I'm just suggesting to think of a scheme where the unused buffer for still image is reused for preview frames when preview is shown.
That will work as well, but it will increase the shot-to-snapshot delay as cache management will need to be performed after the user presses the shot button.
Signature · 3 lines
Regards, Laurent Pinchart