PCIe device driver question
I allocate memory in the user land using memalign function (typically I allocate about 500 MB) and pass this to the kernel space. In my device driver, I call get_user_pages() to lock down the memory and extract the relevant pages. A scatter-gather list is generated using these page addresses and hence derive the dma_addresses using page_to_phys() function. These addresses are programmed into a FIFO in the hardware device using a memory mapped register interface (PCI BAR based). Subsequently the hardware start filling up the pages and interrupt when a block of pages are complete. I notice the hardware hang (PCIe packets don't seem to get the acknowledgements from the root complex) when the DMA address is < 0x0000_0001_0000_0000. I have verified in the hardware that the PCIe packet is created with the correct address as programed by the device driver dma_address. If i can guard some how that the memory allocation is with in a certain area, I can stop the
problem from occuring. Are there any bridge functionality in the Intel architecture that may mask a certain region of memory?
Thanks and regards,
Sanka
----- Original Message ----
From: Robert Hancock <[email protected]>
To: Sanka Piyaratna <[email protected]>
Cc: [email protected]
Sent: Thursday, 31 July, 2008 4:54:48 AM
Subject: Re: PCIe device driver question
Quoted 12 lines · Sanka Piyaratna
Hi,
I am currently developing a PCIe data capture card hardware and the device drivers to drive this. I have implemented DMA on the data capture and the scatter-gather DMA is implemented in the hardware. I am testing this in an
X86_64architecture machine with 4 GB of RAM. I am able to successfully dma data into any memory (dma) address > 0x0000_0001_0000_0000. However, my problem is to dma data to any address less than this. When I try to DMA data to an address less than 0x0000_0001_0000_0000, the hardware device hangs indicating that the address does not exist.
Quoted 5 lines
I have implemented the DMA mask to be full 64 bit and my hardware is capable of transfering data to any address < 8TB. I am using kernel version 2.6.23.11.
Could you please let me know what I might be doing wrong?
The kernel can't do anything to stop you from DMAing anywhere you want (barring the system having special IOMMU hardware). If you overwrite something you shouldn't have you'll cause a crash, but the kernel has no influence on it really.
Unless you're messing up the DMA addresses somehow and writing into a space that's not actually RAM (like the MMIO memory hole or something), my guess is it's likely a hardware problem..
Find a better answer, faster with the new Yahoo!7 Search. www.yahoo7.com.au/search
Quoted 2 lines · Sanka Piyaratna
I allocate memory in the user land using memalign function (typically I allocate about 500 MB) and pass this to the kernel space. In my device driver, I call
get_user_pages()to lock down the memory and extract the relevant pages. A scatter-gather list is generated using these page addresses and hence derive thedma_addressesusingpage_to_phys()function. These addresses are programmed into a FIFO in the hardware device using a memory mapped register interface (PCI BAR based). Subsequently the hardware start filling up the pages and interrupt when a block of pages are complete. I notice the hardware hang (PCIe packets don't seem to get the acknowledgements from the root complex) when the DMA address is < 0x0000_0001_0000_0000. I have verified in the hardware that the PCIe packet is created with the correct address as programed by the device driverdma_address. If i can guard some how that the memory allocation is with in a certain area, I can stop the
problem from occuring. Are there any bridge functionality in the Intel architecture that may mask a certain region of memory?
How are you formatting the addresses in the TLP? The PCI Express spec says that for addresses below 4GB the 32-bit addressing format must be used (you can't use the 64-bit format unconditionally). It could be that is what is making the chipset unhappy.