Discontiguous Memory
We have a client building a PPC8548-based product who insists that we allocate DRAM real address space in two large chunks at 0-2GB and 4-6GB in the 36-bit address space. It doesn't look like U-Boot's bd_info structure allows for that, and it doesn't look like the Linux init routines (which accesses the passed table) knows how to handle discontiguous memory either (as opposed to X86 Linux, which can accept an E820 table).
It looks like Linux cleans out the TLB's pretty quickly, so it wouldn't know the VA-to-RA mapping.
I've seen papers on some PPC Linuxes that handle large discontiguous real DRAM memory spaces, but it doesn't look like the Linux in ELDK does. Is that correct?
This is just the start of a headache between the architects and we poor implementers. It's not clear to us that, even with ATMU support, how normal drivers are going to know how to create real addresses for buffers and such to use when programming DMA controllers or external PCI devices.
Are this really a problem or is there kernel provision for the PPC for discontiguous real memory DRAM space?
Any comments would be VERY appreciated.
Regards,
J
Signature · 2 lines
View this message in context: http://www.nabble.com/Discontiguous-Memory-tf2279691.html#a6332580 Sent from the linuxppc-dev forum at Nabble.com.
Am Friday 15 September 2006 22:22 schrieb jbahr:
Quoted 4 lines
Are this really a problem or is there kernel provision for the PPC for discontiguous real memory DRAM space?
Any comments would be VERY appreciated.
You probably want to enable the CONFIG_ARCH_SPARSEMEM_ENABLE option, which is currently dependant on CONFIG_PPC64.
Look in the early_init_dt_scan_memory function (arch/powerpc/kernel/prom.c) for how to layout the "memory" nodes in your device tree.
Arnd <><
Thanks, Arnd. As we're running a 32-bit 8548, looks like we might not be able to enable sparse memory maps.
Good thought, though.
J
-----Original Message-----
From: Arnd Bergmann [mailto:[email protected]]
Sent: Friday, September 15, 2006 5:25 PM
To: [email protected]
Cc: jbahr
Subject: Re: Discontiguous Memory
Am Friday 15 September 2006 22:22 schrieb jbahr:
Quoted 4 lines
Are this really a problem or is there kernel provision for the PPC for discontiguous real memory DRAM space?
Any comments would be VERY appreciated.
You probably want to enable the CONFIG_ARCH_SPARSEMEM_ENABLE option, which is currently dependant on CONFIG_PPC64.
Look in the early_init_dt_scan_memory function (arch/powerpc/kernel/prom.c) for how to layout the "memory" nodes in your device tree.
Arnd <><
Quoted 19 lines · jbahr
We have a client building a PPC8548-based product who insists that we allocate DRAM real address space in two large chunks at 0-2GB and 4-6GB in the 36-bit address space. It doesn't look like U-Boot's
bd_infostructure allows for that, and it doesn't look like the Linux init routines (which accesses the passed table) knows how to handle discontiguous memory either (as opposed to X86 Linux, which can accept an E820 table).It looks like Linux cleans out the TLB's pretty quickly, so it wouldn't know the VA-to-RA mapping.
I've seen papers on some PPC Linuxes that handle large discontiguous real DRAM memory spaces, but it doesn't look like the Linux in ELDK does. Is that correct?
This is just the start of a headache between the architects and we poor implementers. It's not clear to us that, even with ATMU support, how normal drivers are going to know how to create real addresses for buffers and such to use when programming DMA controllers or external PCI devices.
Quoted 4 lines
Are this really a problem or is there kernel provision for the PPC for discontiguous real memory DRAM space?
Any comments would be VERY appreciated.
There is no support for any of this in the current PowerPC 32 bits kernel. It's possible to add it, though, but it's not alrady there. There are two main issues:
- One is to enable support for sparse memory maps and add the necessary
support to the low mm code. Not terribly hard (see how it's done for 64 bits)
- A more annoying one is support for DMA since a lot of devices cannot
DMA above 32 bits. A lot of 'sane' platforms that provide more memory than can be DMA'ed in 32 bits also provide an IOMMU that does page translation of incoming DMAs. This isn't your case however, thus you would have to implement some of the alternate solutions used on x86_64 (Intel 64 bits platforms also lack an iommu). This essentially consist of defining a ZONE_DMA32 and possibly also using swiotlb to do bounce buffering for drivers that don't deal with ZONE_DMA32 yet.
Ben.
Am Saturday 16 September 2006 02:56 schrieb Benjamin Herrenschmidt:
Quoted 8 lines
=A0- A more annoying one is support for DMA since a lot of devices cannot DMA above 32 bits. A lot of 'sane' platforms that provide more memory than can be DMA'ed in 32 bits also provide an IOMMU that does page translation of incoming DMAs. This isn't your case however, thus you would have to implement some of the alternate solutions used on
x86_64(Intel 64 bits platforms also lack an iommu). This essentially consist of defining aZONE_DMA32and possibly also using swiotlb to do bounce buffering for drivers that don't deal withZONE_DMA32yet.
On 32 bit platforms, you normally have ZONE_NORMAL for up to 4G and ZONE_HIGHMEM beyond, so you don't need an extra ZONE_DMA32.
Arnd <><
Quoted 10 lines · Arnd Bergmann
Am Saturday 16 September 2006 02:56 schrieb Benjamin Herrenschmidt:
Quoted 8 lines
- A more annoying one is support for DMA since a lot of devices cannot
DMA above 32 bits. A lot of 'sane' platforms that provide more memory than can be DMA'ed in 32 bits also provide an IOMMU that does page translation of incoming DMAs. This isn't your case however, thus you would have to implement some of the alternate solutions used onx86_64(Intel 64 bits platforms also lack an iommu). This essentially consist of defining aZONE_DMA32and possibly also using swiotlb to do bounce buffering for drivers that don't deal withZONE_DMA32yet.
Quoted 2 lines
On 32 bit platforms, you normally have
ZONE_NORMALfor up to 4G andZONE_HIGHMEMbeyond, so you don't need an extraZONE_DMA32.
No, in fact, it's even worse :) Your example works only on platforms that can do 4G/4G split. On ppc32, currently, we don't (though we could with some freescale BookE and their new instructions to load/store cross AS), thus we have a 3G/1G split, which limits ZONE_NORMAL to about 768Mb.
So ZONE_HIGHEMEM is actually most of your memory. But you -can- DMA from highmem and can't have memory be both HIGHMEM and ZONE_DMA32 (though I suppose devices requesting ZONE_DMA32 will hit ZONE_NORMAL). It's a mess...
Ben.