Skip to contentSkip to conversation

allocation failed: out of vmalloc space error treating and VIDEO1394 IOC LISTEN CHANNEL ioctl failed problem

DISCUSSION Stefan Richter 8 participants 16 messages Jan 14, 2007 – Jan 16, 2007 2 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#2 lore raw

On 10 Jan, Peter Antoniac wrote:
[...]

Quoted 6 lines

Problem is: how to get the VMALLOC_RESERVED value for the kernel that is running? I couldn't find any standard way to do that (something to apply to GNU Linux and the like). All the things I could get were the default value being 128MiB :) and that is it. Now, I could just put 128, but what if somebody changes that, or in some new distro suddenly decides to make it different? Even worse, what if it is an old kernel with 64 setting?

[...]

Maybe somebody at LKML has answers?

Signature · 3 lines
Stefan Richter
-=====-=-=== ---= -===-
http://arcgraph.de/sr/
#3 lore raw
Quoted 11 lines · Stefan Richter

On 10 Jan, Peter Antoniac wrote:
[...]

Quoted 6 lines

Problem is: how to get the VMALLOC_RESERVED value for the kernel that is running? I couldn't find any standard way to do that (something to apply to GNU Linux and the like). All the things I could get were the default value being 128MiB :) and that is it. Now, I could just put 128, but what if somebody changes that, or in some new distro suddenly decides to make it different? Even worse, what if it is an old kernel with 64 setting?

[...]

Maybe somebody at LKML has answers?

vmalloc space is limited; you really can't assume you can get any more than 64Mb or so (and even then it's thight on some systems already); it really sounds like vmalloc space isn't the right solution for your problem whatever it is (context is lost in the quoted mail)... can you restate the problem to see if there's a better solution possible?

Greetings,
Arjan van de Ven

Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
#4 lore raw
Quoted 2 lines · Arjan van de Ven

vmalloc space is limited; you really can't assume you can get any more than 64Mb or so (and even then it's thight on some systems already);

I suppose "grep VmallocChunk /proc/meminfo" shows what is available?

Quoted 4 lines

it really sounds like vmalloc space isn't the right solution for your problem whatever it is (context is lost in the quoted mail)... can you restate the problem to see if there's a better solution possible?

Thanks. Below is Peter's message to linux1394-devel. The previous discussion went over libdc1394-devel which I don't receive. Obviously he wants a really large buffer for reception of an isochronous stream. I guess his reason is highly application specific...

| Hi all,
| 
| I've been trying to get a resolution to the problem with vmalloc error (the 
| <<allocation failed: out of vmalloc space - use vmalloc=<size> to increase 
| size.>> kernel error message thing). The plan how to resolve the issue is 
| simple; get the buffer size that we try to allocate (vmmap.nb_buffers * 
| vmmap.buf_size) and compare it to the VMALLOC_RESERVED. If too big, error 
| with explanation how to fix it. If small, other error (usual out of mem). 
| Problem is: how to get the VMALLOC_RESERVED value for the kernel that is 
| running? I couldn't find any standard way to do that (something to apply to 
| GNU Linux and the like). All the things I could get were the default value 
| being 128MiB :) and that is it. Now, I could just put 128, but what if 
| somebody changes that, or in some new distro suddenly decides to make it 
| different? Even worse, what if it is an old kernel with 64 setting?
| 
| Currently, in the SVN version, Damien was kind to change so that a message 
| gets printed with a full explanation of how to treat it. Still, this is a 
| compromise solution and not the elegant one that I was looking for. I believe 
| and hope that maybe somebody had this issue before and could help with some 
| suggestions...
| 
| So, my question is: anybody knows the way to get to the kernel value like 
| VMALLOC_RESERVED or something around this area (a function like getpagesize 
| or sysconf)? It will do a great deal to solve the problematic error treatment 
| in the library...
| 
| Thank you.
| 
| For your reference, this is in response to this line of thinking or the 
| libdc1394-devel thread:
| [...]
| > > When I set NUM_BUFFERS (number of DMA buffers) to a value greater than 5
| > > the program dies like this:
| > >
| > > (dc1394_capture.c) VIDEO1394_IOC_LISTEN_CHANNEL ioctl failed!
| > > Libdc1394 error (dc1394_capture.c:dc1394_capture_setup_dma:382): Capture
| > > is not set : Could not setup DMA capture
| [...]
| > > [17723533.496000] video1394_0: Iso receive DMA: 8 buffers of size
| > > 6627328 allocated for a frame size 6624000, each with 1619 prgs
| > > [17723533.516000] video1394_0: iso context 0 listen on channel 1
| > > [17723533.712000] ieee1394: Node [1-01:1023] wants to release broadcast
| > > channel 31.  Ignoring.
| > > [17723534.448000] video1394_1: mask: 0000000000000004 usage:
| > > 0000000000000000
| > > [17723534.448000]
| > > [17723534.508000] video1394_1: Iso receive DMA: 8 buffers of size
| > > 6627328 allocated for a frame size 6624000, each with 1619 prgs
| > > [17723534.532000] video1394_1: iso context 0 listen on channel 2
| > > [17723534.728000] ieee1394: Node [2-01:1023] wants to release broadcast
| > > channel 31.  Ignoring.
| > > [17723535.464000] video1394_2: mask: 0000000000000008 usage:
| > > 0000000000000000
| > > [17723535.464000]
| > > [17723535.464000] printk: 11 messages suppressed.
| > > [17723535.464000] allocation failed: out of vmalloc space - use
| > > vmalloc=<size> to increase size.
| > > [17723535.464000] dma_region_alloc: vmalloc_32() failed
| > > [17723535.464000] video1394_2: Failed to allocate dma buffer
| > > [17723535.464000] video1394_2: Couldn't allocate ir context
| > > [17723535.668000] video1394_0: On release: Iso receive context 0 stop
| > > listening on channel 1
| > > [17723535.676000] video1394_1: On release: Iso receive context 0 stop
| > > listening on channel 2
| [...]
| > ------------------------------
| >
| > Message: 2
| > Date: Fri, 05 Jan 2007 17:30:39 +0100
| > From: Martin Peris <xxxxxxxxxxxxxxxxxxx>
| > Subject: Re: [libdc1394-devel] [SPAM RBL] Re:
| >         VIDEO1394_IOC_LISTEN_CHANNEL    ioctl failed! and Bad images
| > To: Damien Douxchamps <xxxxxxxxxxxxxxxxxxxxx>
| > Cc: libdc1394-devel <xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
| [...]
| > I think I have some answers...
| >
| > I've been investigating a bit, and the  problem with the limit in the
| > size of the allocated DMA buffer  is not so obscure.
| >
| > vmalloc_32() allocate virtually contiguous memory (32bit addressable),
| > the default maximum amount of memory reserved for this depends on each
| > kernel compilation, and in my case it was set to 128Mbytes that's why I
| > had an error if tried to allocate too many buffers.
| >
| >  but at boot time you can specify how much virtually contiguous memory
| > you want with the parameter vmalloc, so if you want about 512Mbytes of
| > memory for vmalloc you should add the parameter vmalloc=536870912 to the
| > line that defines the kernel parameters. (If you use grub there should
| > be a line like this on /boot/grub/menu.lst)
| >
| > kernel          /boot/vmlinuz-2.6.15-27-686 root=/dev/sda2
| > vmalloc=536870912 ro quiet splash
| >
| >
| > That killed my problem with:
| >
| > dc1394_capture.c) VIDEO1394_IOC_LISTEN_CHANNEL ioctl failed!
| > Libdc1394 error (dc1394_capture.c:dc1394_capture_setup_dma:382): Capture
| > is not set : Could not setup DMA capture
| [...]
Signature · 3 lines
Stefan Richter
-=====-=-=== ---= -===-
http://arcgraph.de/sr/
#5 lore raw
Quoted 11 lines · Stefan Richter
Quoted 2 lines · Arjan van de Ven

vmalloc space is limited; you really can't assume you can get any more than 64Mb or so (and even then it's thight on some systems already);

I suppose "grep VmallocChunk /proc/meminfo" shows what is available?

Quoted 4 lines

it really sounds like vmalloc space isn't the right solution for your problem whatever it is (context is lost in the quoted mail)... can you restate the problem to see if there's a better solution possible?

Quoted 4 lines

Thanks. Below is Peter's message to linux1394-devel. The previous discussion went over libdc1394-devel which I don't receive. Obviously he wants a really large buffer for reception of an isochronous stream. I guess his reason is highly application specific...

but why does that even use vmalloc? You can just do a scatter gather thing instead... and keep a list of pages that you're mapping into userspace. vmalloc isn't really a requirement for that....

Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
#6 lore raw
In reply to Stefan Richter
Quoted 5 lines · Stefan Richter
Quoted 2 lines · Arjan van de Ven

vmalloc space is limited; you really can't assume you can get any more than 64Mb or so (and even then it's thight on some systems already);

I suppose "grep VmallocChunk /proc/meminfo" shows what is available?

This is more the answer that I expect. But is there a way, function or constant from libdc that can give you the answer, or you have to get it from the /proc/meminfo? Now, if you can only get it from the /proc/meminfo, is this info always there? As far as I remember, you can compile kernels without the /proc/meminfo...

And yes, probably we would have to get rid of this issue and look for an alternative way. Still, from a technical point of view, I am really curious if this information on the size of VmallocChunk is available to the developer in other forms...

Thank you all!

Cheers,
Peter

Quoted 120 lines
Quoted 4 lines

it really sounds like vmalloc space isn't the right solution for your problem whatever it is (context is lost in the quoted mail)... can you restate the problem to see if there's a better solution possible?

Thanks. Below is Peter's message to linux1394-devel. The previous discussion went over libdc1394-devel which I don't receive. Obviously he wants a really large buffer for reception of an isochronous stream. I guess his reason is highly application specific...

| Hi all,
|
| I've been trying to get a resolution to the problem with vmalloc error
| (the <<allocation failed: out of vmalloc space - use vmalloc=<size> to
| increase size.>> kernel error message thing). The plan how to resolve the
| issue is simple; get the buffer size that we try to allocate
| (vmmap.nb_buffers * vmmap.buf_size) and compare it to the
| VMALLOC_RESERVED. If too big, error with explanation how to fix it. If
| small, other error (usual out of mem). Problem is: how to get the
| VMALLOC_RESERVED value for the kernel that is running? I couldn't find
| any standard way to do that (something to apply to GNU Linux and the
| like). All the things I could get were the default value being 128MiB :)
| and that is it. Now, I could just put 128, but what if somebody changes
| that, or in some new distro suddenly decides to make it different? Even
| worse, what if it is an old kernel with 64 setting?
|
| Currently, in the SVN version, Damien was kind to change so that a
| message gets printed with a full explanation of how to treat it. Still,
| this is a compromise solution and not the elegant one that I was looking
| for. I believe and hope that maybe somebody had this issue before and
| could help with some suggestions...
|
| So, my question is: anybody knows the way to get to the kernel value like
| VMALLOC_RESERVED or something around this area (a function like
| getpagesize or sysconf)? It will do a great deal to solve the problematic
| error treatment in the library...
|
| Thank you.
|
| For your reference, this is in response to this line of thinking or the
| libdc1394-devel thread:
| [...]
|
| > > When I set NUM_BUFFERS (number of DMA buffers) to a value greater
| > > than 5 the program dies like this:
| > >
| > > (dc1394_capture.c) VIDEO1394_IOC_LISTEN_CHANNEL ioctl failed!
| > > Libdc1394 error (dc1394_capture.c:dc1394_capture_setup_dma:382):
| > > Capture is not set : Could not setup DMA capture
|
| [...]
|
| > > [17723533.496000] video1394_0: Iso receive DMA: 8 buffers of size
| > > 6627328 allocated for a frame size 6624000, each with 1619 prgs
| > > [17723533.516000] video1394_0: iso context 0 listen on channel 1
| > > [17723533.712000] ieee1394: Node [1-01:1023] wants to release
| > > broadcast channel 31.  Ignoring.
| > > [17723534.448000] video1394_1: mask: 0000000000000004 usage:
| > > 0000000000000000
| > > [17723534.448000]
| > > [17723534.508000] video1394_1: Iso receive DMA: 8 buffers of size
| > > 6627328 allocated for a frame size 6624000, each with 1619 prgs
| > > [17723534.532000] video1394_1: iso context 0 listen on channel 2
| > > [17723534.728000] ieee1394: Node [2-01:1023] wants to release
| > > broadcast channel 31.  Ignoring.
| > > [17723535.464000] video1394_2: mask: 0000000000000008 usage:
| > > 0000000000000000
| > > [17723535.464000]
| > > [17723535.464000] printk: 11 messages suppressed.
| > > [17723535.464000] allocation failed: out of vmalloc space - use
| > > vmalloc=<size> to increase size.
| > > [17723535.464000] dma_region_alloc: vmalloc_32() failed
| > > [17723535.464000] video1394_2: Failed to allocate dma buffer
| > > [17723535.464000] video1394_2: Couldn't allocate ir context
| > > [17723535.668000] video1394_0: On release: Iso receive context 0 stop
| > > listening on channel 1
| > > [17723535.676000] video1394_1: On release: Iso receive context 0 stop
| > > listening on channel 2
|
| [...]
|
| > ------------------------------
| >
| > Message: 2
| > Date: Fri, 05 Jan 2007 17:30:39 +0100
| > From: Martin Peris <xxxxxxxxxxxxxxxxxxx>
| > Subject: Re: [libdc1394-devel] [SPAM RBL] Re:
| >         VIDEO1394_IOC_LISTEN_CHANNEL    ioctl failed! and Bad images
| > To: Damien Douxchamps <xxxxxxxxxxxxxxxxxxxxx>
| > Cc: libdc1394-devel <xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
|
| [...]
|
| > I think I have some answers...
| >
| > I've been investigating a bit, and the  problem with the limit in the
| > size of the allocated DMA buffer  is not so obscure.
| >
| > vmalloc_32() allocate virtually contiguous memory (32bit addressable),
| > the default maximum amount of memory reserved for this depends on each
| > kernel compilation, and in my case it was set to 128Mbytes that's why I
| > had an error if tried to allocate too many buffers.
| >
| >  but at boot time you can specify how much virtually contiguous memory
| > you want with the parameter vmalloc, so if you want about 512Mbytes of
| > memory for vmalloc you should add the parameter vmalloc=536870912 to
| > the line that defines the kernel parameters. (If you use grub there
| > should be a line like this on /boot/grub/menu.lst)
| >
| > kernel          /boot/vmlinuz-2.6.15-27-686 root=/dev/sda2
| > vmalloc=536870912 ro quiet splash
| >
| >
| > That killed my problem with:
| >
| > dc1394_capture.c) VIDEO1394_IOC_LISTEN_CHANNEL ioctl failed!
| > Libdc1394 error (dc1394_capture.c:dc1394_capture_setup_dma:382):
| > Capture is not set : Could not setup DMA capture
|
| [...]
#7 lore raw
Quoted 2 lines · Peter Antoniac

This is more the answer that I expect. But is there a way, function or constant from **libdc** that can give you the answer, or you have to get it

...
What was I thinking... not from libdc but from LIBC :)

#8 lore raw
In reply to Arjan van de Ven
Quoted 6 lines · Arjan van de Ven
Quoted 5 lines · Stefan Richter

On 10 Jan, Peter Antoniac wrote:
[...]

Quoted 1 line

Problem is: how to get the VMALLOC_RESERVED value for the kernel that is running? I couldn't find any standard way to do that (something to apply to GNU Linux and the like). All the things I could get were the default value being 128MiB :) and that is it. Now, I could just put 128, but what if somebody changes that, or in some new distro suddenly decides to make it different? Even worse, what if it is an old kernel with 64 setting?

[...]

Quoted 9 lines
Quoted 1 line

Maybe somebody at LKML has answers?

vmalloc space is limited; you really can't assume you can get any more than 64Mb or so (and even then it's thight on some systems already); it really sounds like vmalloc space isn't the right solution for your problem whatever it is (context is lost in the quoted mail)... can you restate the problem to see if there's a better solution possible?

I've used vmalloc in the past, and not had a problem, but it is a fair question, how do you find out how much space is available? Other than a binary vmalloc/release loop.

Signature · 3 lines
bill davidsen <[email protected]>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979
#9 lore raw
Quoted 5 lines
Quoted 2 lines

I've used vmalloc in the past, and not had a problem, but it is a fair question, how do you find out how much space is available? Other than a binary vmalloc/release loop.

that's a really hard question since it's dynamic;
I suppose a good rule of thumb is "never expect more than 16Mb to work" (when I said "64Mb" that was for the kernel as a whole, and there's other users than just "you" :)

if you need that much you probably should redesign your algorithms to not need vmalloc in the first place....

Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
#10 lore raw
Quoted 2 lines · Arjan van de Ven

if you need that much you probably should redesign your algorithms to not need vmalloc in the first place....

I think you've convinced me that vmalloc is not a good choice when a driver needs a large buffer (many megabytes) for DMA.

In this case, we need a large ring buffer for reception of isochronous packets from a firewire device. If I understand you correctly, you are suggesting that this buffer be obtained as followed:

1. Application performs malloc() in user-space and mmap()s it.
2. Driver uses vmalloc_to_page() on every page of the malloc'ed memory and constructs a scatter-gather list.
3. Map the sg list with pci_map_sg().
4. Commence DMA.

Is that correct? In particular, does it do the right thing in terms of pinning the memory and dealing with high memory?

I notice that the block I/O API has some convenience functions for this, but this is not a block device. Are there some other convenience functions that can be used?

Forgive me if these are obvious questions -- I'm not the developer of video1394, but I'd still like get it right for the new firewire stack that's being developed.

Thanks,

David

#11 lore raw
Quoted 20 lines · David Moore
Quoted 2 lines · Arjan van de Ven

if you need that much you probably should redesign your algorithms to not need vmalloc in the first place....

I think you've convinced me that vmalloc is not a good choice when a driver needs a large buffer (many megabytes) for DMA.

In this case, we need a large ring buffer for reception of isochronous packets from a firewire device. If I understand you correctly, you are suggesting that this buffer be obtained as followed:

1. Application performs malloc() in user-space and mmap()s it.
2. Driver uses vmalloc_to_page() on every page of the malloc'ed memory and constructs a scatter-gather list.
3. Map the sg list with pci_map_sg().
4. Commence DMA.

Is that correct? In particular, does it do the right thing in terms of pinning the memory and dealing with high memory?

Quoted 7 lines

I notice that the block I/O API has some convenience functions for this, but this is not a block device. Are there some other convenience functions that can be used?

Forgive me if these are obvious questions -- I'm not the developer of video1394, but I'd still like get it right for the new firewire stack that's being developed.

David, thanks for bringing this up. Indeed if vmalloc is not the right way for allocating big buffers, we need to figure something else out. My impression was that the vmalloc group of functions could be used for big allocations since they don't require the underlying memory to be physically contiguous.

What I'm doing currently in the new firewire stack is to vmalloc the memory to be used for isochronous payload and then use remap_vmalloc_range() to map the memory to the user. I never access the contents from the kernel side, I just use vmalloc so I can pass the pointer to remap_vmalloc_range(). Maybe this is overkill and a better way to do this is to call get_page() a number of times and manually add these pages to the process address space without ever setting up a kernel side mapping for these.

However, what I'd really like to do is to leave it to user space to allocate the memory as David describes. In the transmit case, user space allocates memory (malloc or mmap) and loads the payload into that buffer. Then is does an ioctl() on the firewire control device to indicate the location of this buffer, describe how that payload is to be split into packets, and optionally a header per packet to prepend. The kernel side driver then converts the user space addresses to pages, pins the pages in question, and sets up dma programs to transmit the packets.

Likewise for reception, user space allocates buffers for receiving the data and then instructs the kernel the receive a certain amount of data into this buffer. The kernel pins the pages backing the user space buffer and sets up dma to received into those pages. Once a page it full, it's unpinned and userspace is notified.

It's not too difficult from what I'm doing now, I'd just like to give user space more control over the buffers it uses for streaming (i.e. letting user space allocate them). What I'm missing here is: how do I actually pin a page in memory? I'm sure it's not too difficult, but I haven't yet figured it out and I'm sure somebody knows it off the top of his head.

cheers,
Kristian

#12 lore raw
Quoted 5 lines

However, what I'd really like to do is to leave it to user space to allocate the memory as David describes. In the transmit case, user space allocates memory (malloc or mmap) and loads the payload into that buffer.

there is a lot of pain involved with doing things this way, it is a TON better if YOU provide the memory via a custom mmap handler for a device driver.
(there are a lot of security nightmares involved with the opposite model, like the user can put any kind of memory there, even pci mmio space)

Quoted 1 line

Then is does an ioctl() on the firewire control device

ioctls are evil ;) esp an "mmap me" ioctl

Quoted 6 lines

It's not too difficult from what I'm doing now, I'd just like to give user space more control over the buffers it uses for streaming (i.e. letting user space allocate them). What I'm missing here is: how do I actually pin a page in memory? I'm sure it's not too difficult, but I haven't yet figured it out and I'm sure somebody knows it off the top of his head.

again the best way is for you to provide an mmap method... you can then fill in the pages and keep that in some sort of array; this is for example also what the DRI/DRM layer does for textures etc...

Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
#13 lore raw
Quoted 6 lines · Arjan van de Ven
Quoted 5 lines

However, what I'd really like to do is to leave it to user space to allocate the memory as David describes. In the transmit case, user space allocates memory (malloc or mmap) and loads the payload into that buffer.

Quoted 6 lines

there is a lot of pain involved with doing things this way, it is a TON better if YOU provide the memory via a custom mmap handler for a device driver.
(there are a lot of security nightmares involved with the opposite model, like the user can put any kind of memory there, even pci mmio space)

OK, point taken. I don't have a strong preference for the opposite model, it just seems elegant that you can let user space handle allocation and pin and map the pages as needed. But you're right, it certainly is easier to give safe memory to user space in the first place rather than try to make sure user space isn't trying to trick us.

Quoted 3 lines
Quoted 1 line

Then is does an ioctl() on the firewire control device

ioctls are evil ;) esp an "mmap me" ioctl

Ah, I'm not mmap'ing it from the ioctl, I do implement the mma file operation for this. However, you have to do an ioctl before mapping the device to configure the dma context.

Other than that what is the problem with ioctls, and more interesting, what is the alternative? I don't expect (or want) a bunch of syscalls to be added for this, so I don't really see what other mechanism I should use for this.

Quoted 10 lines
Quoted 6 lines

It's not too difficult from what I'm doing now, I'd just like to give user space more control over the buffers it uses for streaming (i.e. letting user space allocate them). What I'm missing here is: how do I actually pin a page in memory? I'm sure it's not too difficult, but I haven't yet figured it out and I'm sure somebody knows it off the top of his head.

again the best way is for you to provide an mmap method... you can then fill in the pages and keep that in some sort of array; this is for example also what the DRI/DRM layer does for textures etc...

That sounds a lot like what I have now (mmap method, array of pages) so I'll just stick with that.

thanks,
Kristian

#14 lore raw
Quoted 8 lines · Kristian Høgsberg
Quoted 6 lines · Arjan van de Ven

there is a lot of pain involved with doing things this way, it is a TON better if YOU provide the memory via a custom mmap handler for a device driver.
(there are a lot of security nightmares involved with the opposite model, like the user can put any kind of memory there, even pci mmio space)

Quoted 6 lines

OK, point taken. I don't have a strong preference for the opposite model, it just seems elegant that you can let user space handle allocation and pin and map the pages as needed. But you're right, it certainly is easier to give safe memory to user space in the first place rather than try to make sure user space isn't trying to trick us.

I am glad that the discussion is heading to the right place thanks to David.

Yes. Probably that is the best solution. In the case of the ring buffers, based on my discussion with Damien, 4 buffers are probably optimal. If the user is allocating them, in case of normal cameras, this is somewhere around 4 MiB, lets say maxim 16 MiB. So, everything should be ok for normal people, at least for now. The problem is when the cameras require bigger images (we are thinking about the future, right) and maybe also more buffers in the DMA ring buffer. If you leave that to the user, it will require some hacking skills if we are using the current model from libdc1394 and video1394. Why? Because if you use 10 buffers with some big images it is likely you are going out of the 64 MiB. In that case, we were thinking to give a nice error (that is why we needed to know the amount available for mmap/vmalloc) and instruct the user to change the kernel boot time allocation of memory in a way that will fit the range (the vmalloc=xxx at startup - the "hacking"). So, in a way, it will be nice to have the solution close to the one proposed by David. Do you think that if the user allocates small buffers (instead of the big ring buffer) and sends the list to the driver, this will help in breaking the 64 limit? I have doubts about it, but I am not good at this level of VMA. Anyway, I hope that something can be done to allow bigger DMA ring buffers without the user needing to reboot the system with some parameter.

Quoted 29 lines
Quoted 3 lines
Quoted 1 line

Then is does an ioctl() on the firewire control device

ioctls are evil ;) esp an "mmap me" ioctl

Ah, I'm not mmap'ing it from the ioctl, I do implement the mma file operation for this. However, you have to do an ioctl before mapping the device to configure the dma context.

Other than that what is the problem with ioctls, and more interesting, what is the alternative? I don't expect (or want) a bunch of syscalls to be added for this, so I don't really see what other mechanism I should use for this.

Quoted 10 lines
Quoted 6 lines

It's not too difficult from what I'm doing now, I'd just like to give user space more control over the buffers it uses for streaming (i.e. letting user space allocate them). What I'm missing here is: how do I actually pin a page in memory? I'm sure it's not too difficult, but I haven't yet figured it out and I'm sure somebody knows it off the top of his head.

again the best way is for you to provide an mmap method... you can then fill in the pages and keep that in some sort of array; this is for example also what the DRI/DRM layer does for textures etc...

That sounds a lot like what I have now (mmap method, array of pages) so I'll just stick with that.

thanks,
Kristian

#15 lore raw
In reply to Kristian Høgsberg
Quoted 7 lines · Kristian Høgsberg
Quoted 3 lines · Arjan van de Ven

again the best way is for you to provide an mmap method... you can then fill in the pages and keep that in some sort of array; this is for example also what the DRI/DRM layer does for textures etc...

That sounds a lot like what I have now (mmap method, array of pages) so I'll just stick with that.

It sounds like the distinction Arjan is getting at is that the buffer should exist in the process's virtual address space instead of the kernel's virtual address space so that we have plenty of space available to us.

Thus, we should use get_user_pages() instead of vmalloc(). I think get_user_pages() will also automatically pin the memory. And we'll also need to call get_user_pages() from a custom mmap() handler so that we know what process virtual address to assign to the region.

Is that right Arjan?

Thanks,

David

#16 lore raw
Quoted 6 lines · David Moore
Quoted 5 lines · Kristian Høgsberg
Quoted 3 lines · Arjan van de Ven

again the best way is for you to provide an mmap method... you can then fill in the pages and keep that in some sort of array; this is for example also what the DRI/DRM layer does for textures etc...

Quoted 7 lines
Quoted 2 lines

That sounds a lot like what I have now (mmap method, array of pages) so I'll just stick with that.

It sounds like the distinction Arjan is getting at is that the buffer should exist in the process's virtual address space instead of the kernel's virtual address space so that we have plenty of space available to us.

exactly! and either way the user would need that space allocated ANYWAY.

Quoted 3 lines

Thus, we should use get_user_pages() instead of vmalloc(). I think get_user_pages() will also automatically pin the memory.

actually if you provide the mmap method yourself you're not going to need get_user_pages(), because it's kernel allocated memory already!

Quoted 3 lines

And we'll also
need to call get_user_pages() from a custom mmap() handler so that we know what process virtual address to assign to the region.

see above; it's one or the other. Personally I'd think the mmap method is simpler, because there's less conditions to consider (again, the malicious user passing you memory that is mmap'd PCI MMIO space is a "fun" example, but there's a lot more cafes which are... funky)

Greetings,
Arjan van de Ven

Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
#17 lore raw
In reply to David Moore

Hi,

Quoted 2 lines

I think you've convinced me that vmalloc is not a good choice when a driver needs a large buffer (many megabytes) for DMA.

Yep. bttv used to do that long time ago, and some people used to run into trouble because the nvidia driver used lots of vmalloc address space ...

Quoted 9 lines

In this case, we need a large ring buffer for reception of isochronous packets from a firewire device. If I understand you correctly, you are suggesting that this buffer be obtained as followed:

1. Application performs malloc() in user-space and mmap()s it.
2. Driver uses vmalloc_to_page() on every page of the malloc'ed memory and constructs a scatter-gather list.
3. Map the sg list with pci_map_sg().
4. Commence DMA.

video4linux drivers do this:

(1) app opens /dev/video0
(2) app issues some ioctls to set buffer count, format and size
(3) app uses mmap(/dev/video0) to map the buffers.

Buffer memory is simple userspace memory, it is allocated by the nopage handler. Memory is pinned down by get_user_pages().

Quoted 2 lines

Are there some other convenience
functions that can be used?

Look at drivers/media/video/video-buf.c, that is the buffer management code shared by v4l drivers. It should at least give an idea how it works, not sure you can actually reuse the code. Chances are not too bad though, it is used not only for video buffers, but also for audio and mpeg2 data, so it could be generic enough to fit your needs too.

cheers,
Gerd

Signature · 1 line
Gerd Hoffmann <[email protected]>