Skip to contentSkip to conversation

DMA transfer with kiobuf, kernel 2.4.21

DISCUSSION sej 2 participants 5 messages Nov 15, 2005 – Nov 16, 2005

lore
Share & tools
lore

sej

author
#0 lore raw

Hi,
I allocate a big chunck of memory from user space with :

#define MEM_SIZE_DMA (128*1024*1024)
// allocate 128MB of memory
void *_pVirtualMem = memalign(sysconf(_SC_PAGESIZE), MEM_SIZE_DMA);

// Reserve memory
memset(_pVirtualMem, 0, MEM_SIZE_DMA);

// Lock memory
if (!mlock(_pVirtualMem, MEM_SIZE_DMA ))
{
free(_pVirtualMem);
return false;
}

Then I call an IOCTL from my driver (DmaMapDescrpImg) to create a DMA scatter gather list.
This solution works when I have 1GB of memory. If I put 2GB of memory, it won't work because I can't resolve the physical address of my User memory area.

My configuration is a Redhat enterprise 3.3 SMP on Xeon 3.2GHz. The kernel is configured for 64GB support.
So the kernel only have access to LOW part of memory.
How can I force my User allocation in LOW part, or how can I replace map_user_from_kiobuf() to support this memory ?
With kiobuf, I make DMA scatter gather list at the opening of the card and then I use it to perform DMA transfer. With get_user_pages(), I must make a scatter gather list for each transfer, and make pages dirty after each transfer ?
I am not sure that get_user_pages() is the good solution.
Is there another method to perform my DMA transfer.
I must allocate my memory in user space, and I can't copy it from kernel to user because I'll have a big overhead.

Regards.
Sebastien

Kernel IOCTL :

int DmaMapDescrpImg (int board_num, u32 arg)
{
struct kiobuf *iobuf;

// get parameters
get_user ( ... );

// Create kiovec into decriptor pointer registers
// Map kiovec into descriptor pointer register
result = alloc_kiovec(1, &iobuf);
if (result)
{
TRACE_ERR("alloc_kiovec failed\n");
return 1;
}

result = map_user_kiobuf(READ,
iobuf, // struct kiobuf
arg, // user addr (buffer passed from user)
iSize);// size
if (result)
{
TRACE_ERR("map_user_kiobuf failed\n");
free_kiovec(1, &iobuf);
return 1;
}

// ************************* Build descriptors
transfer = &(pdx[board_num].ReadTransfer);

// Number of descriptor needed
nbDescrpImg = iSize/PAGE_SIZE + ((iSize%PAGE_SIZE) ? 1 : 0); // Index of the first descriptor in transfer->Descript[] idxDescrpImg = nImage * nbDescrpImg;

transfer->image[nImage].Descript = &(transfer->Descript[idxDescrpImg]); transfer->imageInfo->flag[nImage] = IMG_UNUSED; //IMG_UNLOCK;

for(i=idxDescrpImg, idxIobuf = 0 ; i < idxDescrpImg + nbDescrpImg - 1 ; i++, idxIobuf++)
{
transfer->Descript[i].size = PAGE_SIZE; transfer->Descript[i].pciaddr = (ULONG)
virt_to_phys(page_address(iobuf->maplist[idxIobuf])); transfer->Descript[i].localaddr = localaddr; transfer->Descript[i].next = (ULONG) virt_to_phys(&(transfer->Descript[i+1])) | PLX_PCI | PLX_READ;

}

// End Of Chain

transfer->Descript[i].size = PAGE_SIZE;
transfer->Descript[i].pciaddr = (ULONG)
virt_to_phys(page_address(iobuf->maplist[idxIobuf]));
transfer->Descript[i].localaddr = localaddr;
transfer->Descript[i].next = PLX_PCI | (bRead ? PLX_READ :
PLX_WRITE)
| PLX_EOC;

// ******************************************
// unmap and free kiobuf
unmap_kiobuf(iobuf);
free_kiovec(1, &iobuf);
return 0;
}

#1 lore raw
Quoted 17 lines · sej

Hi,
I allocate a big chunck of memory from user space with :

#define MEM_SIZE_DMA (128*1024*1024)
// allocate 128MB of memory
void *_pVirtualMem = memalign(sysconf(_SC_PAGESIZE), MEM_SIZE_DMA);

// Reserve memory
memset(_pVirtualMem, 0, MEM_SIZE_DMA);

// Lock memory
if (!mlock(_pVirtualMem, MEM_SIZE_DMA ))
{
free(_pVirtualMem);
return false;
}

Quoted 2 lines

Then I call an IOCTL from my driver (DmaMapDescrpImg) to create a DMA scatter gather list.

that sounds the wrong approach.. why don't you make your device driver export an mmap function.. and let the userspace app use that ?

Quoted 4 lines

transfer->Descript[i].size = PAGE_SIZE;
transfer->Descript[i].pciaddr = (ULONG)
virt_to_phys(page_address(iobuf->maplist[idxIobuf]));

you really need to use the PCI DMA mapping api!

author
#2 lore raw
Quoted 2 lines

that sounds the wrong approach.. why don't you make your device driver export an mmap function.. and let the userspace app use that ?

I can't because I need to allocate 128MB of memory per PCI card and if I put for example 4 cards, I'll have 512MB in kernel memory, and I think there will be some problem in kernel.

Quoted 7 lines

transfer->Descript[i].size = PAGE_SIZE;
transfer->Descript[i].pciaddr = (ULONG)
virt_to_phys(page_address(iobuf->maplist[idxIobuf]));

you really need to use the PCI DMA mapping api!

I have a plx bridge PCI9656 with a DMA controler. So I have to make a descriptor table with physical address and size. I work in 32 bits address mode, but I don't know which function to call to get a 36bits address for my controler.

Regards.
Sebastien

Quoted 45 lines · sej
Quoted 19 lines

Hi,
I allocate a big chunck of memory from user space with :

#define MEM_SIZE_DMA (128*1024*1024)
// allocate 128MB of memory
void *_pVirtualMem = memalign(sysconf(_SC_PAGESIZE), MEM_SIZE_DMA);

// Reserve memory
memset(_pVirtualMem, 0, MEM_SIZE_DMA);

// Lock memory
if (!mlock(_pVirtualMem, MEM_SIZE_DMA ))
{
free(_pVirtualMem);
return false;
}

Quoted 4 lines

Then I call an IOCTL from my driver (DmaMapDescrpImg) to create a DMA scatter gather list.

that sounds the wrong approach.. why don't you make your device driver export an mmap function.. and let the userspace app use that ?

Quoted 5 lines

transfer->Descript[i].size = PAGE_SIZE;
transfer->Descript[i].pciaddr = (ULONG)
virt_to_phys(page_address(iobuf->maplist[idxIobuf]));

you really need to use the PCI DMA mapping api!

#3 lore raw
Quoted 4 lines · sej
Quoted 2 lines

that sounds the wrong approach.. why don't you make your device driver export an mmap function.. and let the userspace app use that ?

I can't because I need to allocate 128MB of memory per PCI card and if I put for example 4 cards, I'll have 512MB in kernel memory, and I think there will be some problem in kernel.

no there isn't.. there is no rule that memory you allocate for this as to be lowmem... at all.

Quoted 9 lines
Quoted 8 lines

transfer->Descript[i].size = PAGE_SIZE;
transfer->Descript[i].pciaddr = (ULONG)
virt_to_phys(page_address(iobuf->maplist[idxIobuf]));

Quoted 6 lines
Quoted 1 line

you really need to use the PCI DMA mapping api!

I have a plx bridge PCI9656 with a DMA controler. So I have to make a descriptor table with physical address and size.
I work in 32 bits address mode, but I don't know which function to call to get a 36bits address for my controler.

see the PCI DMA mapping api. the docs for it are in Documentation/

author
#4 lore raw

Hi,
thank you for your help.
I will read the DMA documentation. But I need to make DMA on User memory space allocated in user space. Because I don't want the kernel to make allocation and deallocation during execution for stability. Best regards.
Sebastien

Quoted 47 lines · Arjan van de Ven
Quoted 7 lines · sej
Quoted 4 lines

that sounds the wrong approach.. why don't you make your device driver export an mmap function.. and let the userspace app use that ?

I can't because I need to allocate 128MB of memory per PCI card and if I put for example 4 cards, I'll have 512MB in kernel memory, and I think there will be some problem in kernel.

no there isn't.. there is no rule that memory you allocate for this as to be lowmem... at all.

Quoted 10 lines
Quoted 10 lines

transfer->Descript[i].size = PAGE_SIZE;
transfer->Descript[i].pciaddr = (ULONG)
virt_to_phys(page_address(iobuf->maplist[idxIobuf]));

Quoted 7 lines
Quoted 3 lines

you really need to use the PCI DMA mapping api!

I have a plx bridge PCI9656 with a DMA controler. So I have to make a descriptor table with physical address and size. I work in 32 bits address mode, but I don't know which function to call to get a 36bits address for my controler.

see the PCI DMA mapping api. the docs for it are in Documentation/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/