Skip to contentSkip to conversation

Mapping huge user buffers for DMA

DISCUSSION Stephen Williams 3 participants 4 messages Aug 30, 2005 – Aug 31, 2005

lore
Share & tools
lore

Stephen Williams

author
#0 lore raw

I have a PPC405GPr system with an image processing device, that is creating potentially huge amounts of data. In one setup I have a 256Meg system, and I'm trying to map a 192Meg destination buffer using map_user_kiovec and an array of kiobufs.

I'm finding, however, that I'm getting an Oops in map_user_kiovec when it tries this, and I'm wondering where I need to look for any limits I might be overrunning.

Also, I've been considering skipping kiobufs all together and instead using code like this (lifted from map_user_kiobuf)

	/* Try to fault in all of the necessary pages */
	down_read(&mm->mmap_sem);
	/* rw==READ means read from disk, write into memory area */
	err = get_user_pages(current, mm, va, pgcount,
			(rw==READ), 0, iobuf->maplist, NULL);
	up_read(&mm->mmap_sem);

to get the user pages directly. This is really what I want, and I do not need the other functionality of kiobufs. Is the get_user_pages function kosher for use by drivers? Is there a limit to what get_user_pages may map?

Signature · 4 lines
Steve Williams                "The woods are lovely, dark and deep.
steve at icarus.com           But I have promises to keep,
http://www.icarus.com         and lines to code before I sleep,
http://www.picturel.com       And lines to code before I sleep."
#1 lore raw
Quoted 4 lines · Stephen Williams

I have a PPC405GPr system with an image processing device, that is creating potentially huge amounts of data. In one setup I have a 256Meg system, and I'm trying to map a 192Meg destination buffer using map_user_kiovec and an array of kiobufs.

Never tried to map that amount at once. Our code looks, or looked, like this. Found some code... /usr/src/linuxppc_2_4_devel/arch/cris/drivers/examples/kiobuftest.c

Quoted 4 lines

I'm finding, however, that I'm getting an Oops in map_user_kiovec when it tries this, and I'm wondering where I need to look for any limits I might be overrunning.

What does the Oops say??
Hmm... How much space will the vector itself require?

Quoted 11 lines

Also, I've been considering skipping kiobufs all together and instead using code like this (lifted from map_user_kiobuf)

	/* Try to fault in all of the necessary pages */
	down_read(&mm->mmap_sem);
	/* rw==READ means read from disk, write into memory area */
	err = get_user_pages(current, mm, va, pgcount,
			(rw==READ), 0, iobuf->maplist, NULL);
	up_read(&mm->mmap_sem);
Quoted 4 lines

to get the user pages directly. This is really what I want, and I do not need the other functionality of kiobufs. Is the get_user_pages function kosher for use by drivers? Is there a limit to what get_user_pages may map?

Isn't this the way it is done in 2.6 anyway?

/RogerL

author
#2 lore raw
Quoted 11 lines · Roger Larsson
Quoted 4 lines · Stephen Williams

I have a PPC405GPr system with an image processing device, that is creating potentially huge amounts of data. In one setup I have a 256Meg system, and I'm trying to map a 192Meg destination buffer using map_user_kiovec and an array of kiobufs.

Never tried to map that amount at once. Our code looks, or looked, like this. Found some code... /usr/src/linuxppc_2_4_devel/arch/cris/drivers/examples/kiobuftest.c

That's similar to where I'm at now, and I'm finding it inadequate.

Quoted 7 lines
Quoted 3 lines

I'm finding, however, that I'm getting an Oops in map_user_kiovec when it tries this, and I'm wondering where I need to look for any limits I might be overrunning.

What does the Oops say??
Hmm... How much space will the vector itself require?

If placed into a single kiobuf, it would need 49,152 page pointers, which would be 192K bytes for the maplist. Since the kernel uses kmalloc to allocate the maplist, that won't work. Thus, I've created an array of kiobufs to hold a smaller chunk of the pages for the user buffer.

The Oops is a sig11 within the map_user_kiovec. (?!)

I also see errors from the kernel being unable to allocate pages, so it seems like something is not handling an OOM situation very well.

Quoted 11 lines
Quoted 11 lines

Also, I've been considering skipping kiobufs all together and instead using code like this (lifted from map_user_kiobuf)

	/* Try to fault in all of the necessary pages */
	down_read(&mm->mmap_sem);
	/* rw==READ means read from disk, write into memory area */
	err = get_user_pages(current, mm, va, pgcount,
			(rw==READ), 0, iobuf->maplist, NULL);
	up_read(&mm->mmap_sem);
Quoted 7 lines
Quoted 4 lines

to get the user pages directly. This is really what I want, and I do not need the other functionality of kiobufs. Is the get_user_pages function kosher for use by drivers? Is there a limit to what get_user_pages may map?

Isn't this the way it is done in 2.6 anyway?

Well that's how the map_user_kiovec in 2.4 does it. I'm just wondering if I were to do it myself using get_user_pages, would I be peeking around the curtain at interfaces not meant to be public. I'm thinking I need to ues get_user_pages myself to avoid the extra overhead of kiobufs, and I can use vmalloc'ed memory to hold those page pointers for later unmap so I don't run up against the kmalloc limit.

Signature · 4 lines
Steve Williams                "The woods are lovely, dark and deep.
steve at icarus.com           But I have promises to keep,
http://www.icarus.com         and lines to code before I sleep,
http://www.picturel.com       And lines to code before I sleep."
#3 lore raw
In reply to Stephen Williams

Hello, Stephen!

Please have a look at the whole thread around my posts at:
http://lkml.org/lkml/2005/8/4/201

I am working on an mpc8540 (e500) ppc and we are pushing data from the local bus directly to the userspace mlock()ed memory with the dma engine, using scatter/gather (chaining and page/chunk-defragmentation/compression) transfers.

Currently, I send the application's user virtual and mlock()ed address to the kernel driver through an ioctl and do an iopa() at the user virtual addresses (pages) to get to the physical addresses (pages). The system works stable and we can fill up the whole memory (192MByte) with currently 144MByte/s (still improvements possible).

If you are interested in our design, feel free to contact me directly.

Greets,

Clemens Koller
_______________________________
R&D Imaging Devices
Anagramm GmbH
Rupert-Mayer-Str. 45/1
81379 Muenchen
Germany

http://www.anagramm.de
Phone: +49-89-741518-50
Fax: +49-89-741518-19

Quoted 26 lines · Stephen Williams

I have a PPC405GPr system with an image processing device, that is creating potentially huge amounts of data. In one setup I have a 256Meg system, and I'm trying to map a 192Meg destination buffer using map_user_kiovec and an array of kiobufs.

I'm finding, however, that I'm getting an Oops in map_user_kiovec when it tries this, and I'm wondering where I need to look for any limits I might be overrunning.

Also, I've been considering skipping kiobufs all together and instead using code like this (lifted from map_user_kiobuf)

    /* Try to fault in all of the necessary pages */
    down_read(&mm->mmap_sem);
    /* rw==READ means read from disk, write into memory area */
    err = get_user_pages(current, mm, va, pgcount,
            (rw==READ), 0, iobuf->maplist, NULL);
    up_read(&mm->mmap_sem);

to get the user pages directly. This is really what I want, and I do not need the other functionality of kiobufs. Is the get_user_pages function kosher for use by drivers? Is there a limit to what get_user_pages may map?