PREP memory layout and PCI transactions
Hi,
I'm currently porting a third-party RAID driver from ix86 to LinuxPPC. The target is an MTX board with 32 Mbytes of RAM, running a 2.2.10 kernel.
The driver requests the device to fill some memory locations in kernel space via DMA. This works just fine when such memory locations come from pages allocated by get_free_page() or kmalloc().
However, this fails when these locations come from the *static* data of the module, or from pages allocated by vmalloc().
Specifically, the pages returned by get_free_page() or kmalloc() are located around kernel virtual addresses C1xxxxxx (bus addresses 81xxxxxx), while the static data in question and the pages returned by vmalloc() are located behind virtual addresses C4xxxxxx (bus addresses 84xxxxxx).
The PCI analyser sees on the PCI bus a memory write transaction initiated by the device to such an 84xxxxxx address in memory. We can observe that this transaction is never completed because the DEVSEL signal on the bus is not asserted during the correct 5 clock time interval. The transaction is then terminated as with a master-abort termination, meaning that this address is not known by the memory controler as an address that can be accessed from the PCI bus.
Is this a feature, a known restriction, or a bug? Maybe a register describing the accessible address range from the bus is not correctly initialised into the Falcon chip?
Thanks for your attention and time,
--Christophe
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
Quoted 11 lines · Christophe Lizzi
Hi,
I'm currently porting a third-party RAID driver from ix86 to LinuxPPC. The target is an MTX board with 32 Mbytes of RAM, running a 2.2.10 kernel.
The driver requests the device to fill some memory locations in kernel space via DMA. This works just fine when such memory locations come from pages allocated by
get_free_page()orkmalloc().However, this fails when these locations come from the *static* data of the module, or from pages allocated by
vmalloc().
This is normal, all physical memory is mapped starting from 0xc0000000 in the kernel virtual address space. [I think that copying blindly the x86 setup is a mistake but it's not the time to discuss this ;-)]
But module code and data are allocated with vmalloc, and te fixed offset between physical and kernel virtual addresses no more hold (obviously). So you can only do DMA to memory acquired by kmalloc (and incidentally to local buffers on the stack but don't forget that the size is very limited so I recommend against unless you have a very strong argument in favor).
It is possible the the raw io patches which are in recent kernels allow for DMA to vmalloced memory but I did not study them and I'm not sure.
Quoted 4 lines
Specifically, the pages returned by
get_free_page()orkmalloc()are located around kernel virtual addresses C1xxxxxx (bus addresses 81xxxxxx), while the static data in question and the pages returned byvmalloc()are located behind virtual addresses C4xxxxxx (bus addresses 84xxxxxx).
Bus address 84xxxxxx will not point to your physical memory and you'll end up with some weird error on the PCI bus.
Quoted 7 lines
The PCI analyser sees on the PCI bus a memory write transaction initiated by the device to such an 84xxxxxx address in memory. We can observe that this transaction is never completed because the DEVSEL signal on the bus is not asserted during the correct 5 clock time interval. The transaction is then terminated as with a master-abort termination, meaning that this address is not known by the memory controler as an address that can be accessed from the PCI bus.
Normal....
Quoted 1 line
Is this a feature, a known restriction, or a bug? Maybe a register
It is a very well known and documented restriction, which is a problem if your device can't do scatter gather since the size of buffers you can allocate with kmalloc is limited.
Quoted 2 lines
describing the accessible address range from the bus is not correctly initialised into the Falcon chip?
No, oh and by the way if you want to use the Falcon/Raven/Hawk in CHRP mode, grab my patches at:
ftp://vlab1.iram.es/linux-2.[23]
they work on my MVME26xx and 24xx, but the MTX is quite similar AFAICT.
Gabriel.
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
Hello there,
Quoted 6 lines
The driver requests the device to fill some memory locations in kernel space via DMA. This works just fine when such memory locations come from pages allocated by
get_free_page()orkmalloc().However, this fails when these locations come from the *static* data of the module, or from pages allocated by
vmalloc().
This is expected behaviour, since the way the MM code allocates memory is different between kmalloc()'ed and vmalloc()'ed memory.
kmalloc() returns a memory object consiting of a number of consecutive pages of RAM, adressed through the existing memory translation from (all RAM) to (kernel virtual memory). The full physical RAM is mapped starting at 0xc0000000, and kmalloc will return an address within that range.
However, vmalloc() returns a memory object consisting of pages that are only consecutive in kernel virtual address space, but which is composed of individual pages retreived through get_free_pages(). Therefore, these pages are not necessarily consecutive in physical RAM, and the full object is mapped by a new translation to virtual addresses _above_ physical RAM. As a result, the simple offset calculation between kernel virtual address and physical address doesn't work here.
Quoted 4 lines
Specifically, the pages returned by
get_free_page()orkmalloc()are located around kernel virtual addresses C1xxxxxx (bus addresses 81xxxxxx), while the static data in question and the pages returned byvmalloc()are located behind virtual addresses C4xxxxxx (bus addresses 84xxxxxx).
But bus address 0x84xxxxxx doesn't correspond to any existing RAM.
For DMA, you need to use kmalloc(), or assemble your buffer the dirty way through successive calls to get_free_pages() and other wizardry.
Michel
____________________________________________________________________
.sig home alone
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/